PaperlayerView exact revision on arXiv

Source-linked AI summary

The Anatomy of a Personal Health Agent

A. Ali Heydari, Ken Gu, Vidya Srinivas, Hong Yu, Zhihan Zhang, Yuwei Zhang, Akshay Paruchuri, Qian He, Hamid Palangi, Nova Hammerquist, Ahmed A. Metwally, Brent Winslow, Yubin Kim, Kumar Ayush, Yuzhe Yang, Girish Narayanswamy, Maxwell A. Xu, Jake Garrison, Amy Armento Lee, Jenny Vafeiadou, Ben Graef, Isaac R. Galatzer-Levy, Erik Schenck, Andrew Barakat, Javier Perez, Jacqueline Shreibati, John Hernandez, Anthony Z. Faranesh, Javier L. Prieto, Connor Heneghan, Yun Liu, Jiening Zhan, Mark Malhotra, Shwetak Patel, Tim Althoff, Xin Liu, Daniel McDuff, Xuhai "Orson" Xu

arXiv:2508.20148v2cs.AIcs.HCcs.MA

TL;DR

Personal health agents must handle open-ended, personalized health questions, yet no systematic benchmark evaluates this capability in personal health. The paper develops PHA, a multi-agent framework, and evaluates its specialized agents and integrated system across benchmark tasks, demonstrating advantages over baseline solutions.

  • Problem

    Whether LLM-based agents can interpret, decompose, and operationalize open-ended, underspecified, personalized health inquiries remains unclear, with no systematic benchmark for personal-health analysis planning.

  • Method

    PHA orchestrates Data Science, Domain Expert, and Health Coach sub-agents to provide contextualized, personalized, evidence-based health responses.

  • Results

    Across 10 benchmark tasks and more than 1,100 hours of human evaluation involving end-users and health experts, the individual agents and PHA outperformed baseline solutions.

  • Takeaways & Limitations

    The comprehensive multi-agent architecture and holistic evaluation framework establish a concrete foundation for developing accessible personal health agents.

  • Takeaways & Limitations

    The evaluation demonstrates short-term effectiveness but does not yet assess long-term behavior change through multi-session interactions.

Abstract

from arXiv · show

Health is a fundamental pillar of human wellness, and the rapid advancements in large language models (LLMs) have driven the development of a new generation of health agents. However, the application of health agents to fulfill the diverse needs of individuals in daily non-clinical settings is underexplored. In this work, we aim to build a comprehensive personal health agent that is able to reason about multimodal data from everyday consumer wellness devices and common personal health records, and provide personalized health recommendations. To understand end-users' needs when interacting with such an assistant, we conducted an in-depth analysis of web search and health forum queries, alongside qualitative insights from users and health experts gathered through a user-centered design process. Based on these findings, we identified three major categories of consumer health needs, each of which is supported by a specialist sub-agent: (1) a data science agent that analyzes personal time-series wearable and health record data, (2) a health domain expert agent that integrates users' health and contextual data to generate accurate, personalized insights, and (3) a health coach agent that synthesizes data insights, guiding users using a specified psychological strategy and tracking users' progress. Furthermore, we propose and develop the Personal Health Agent (PHA), a multi-agent framework that enables dynamic, personalized interactions to address individual health needs. To evaluate each sub-agent and the multi-agent system, we conducted automated and human evaluations across 10 benchmark tasks, involving more than 7,000 annotations and 1,100 hours of effort from health experts and end-users. Our work represents the most comprehensive evaluation of a health agent to date and establishes a strong foundation towards the futuristic vision of a personal health agent accessible to everyone.

1. Introduction

The paper addresses underexplored daily, non-clinical health-agent needs by combining user-centered requirements analysis with a multi-agent Personal Health Agent (PHA). PHA coordinates Data Science, Domain Expert, and Health Coach agents and is evaluated through a broad framework.

  • Motivation: Everyday consumer health agents must support diverse needs spanning personal data analysis, contextual health interpretation, and personalized actionable guidance.The paper motivates this scope using wearable data, health records, and varied user queries.
  • Requirements: User-centered analysis of online queries, surveys, and expert design input identifies four critical query categories: health knowledge, personal data, wellness advice, and symptoms.These categories cover general wellness, fitness, and medical topics.
  • Specialist roles: The Data Science Agent analyzes personal health data, the Domain Expert Agent contextualizes medical and wearable information, and the Health Coach Agent applies psychological strategies to support goals and behavior change.The Health Coach Agent uses approaches such as motivational interviewing to help users identify barriers and develop plans.
  • Architecture: The PHA coordinates specialist Data Science, Domain Expert, and Health Coach agents to provide integrated, context-aware support for queries requiring multiple capabilities.For example, sleep questions can combine wearable-data analysis, evidence-based interpretation, and personalized coaching.
  • Scope: The work presents PHA as a research foundation for accessible personal health agents rather than a replacement for human health providers or a currently available product.The paper frames the system as foundational research exploring consumer empowerment and holistic health insights.
  • Evaluation: The evaluation assesses individual sub-agent capabilities and the integrated system across 10 benchmark tasks using more than 7,000 annotations and over 1,100 hours of effort.The framework evaluates both component competencies and overall multi-agent efficacy.

2. A User-Centered Design of Personal Health Agent Requirements

The user-centered design process combines naturalistic queries, surveys, and expert synthesis to identify representative personal health-agent requirements. It organizes these requirements into four broad, overlapping critical user-journey categories.

  • Design process: The requirements process integrates real-world user queries, targeted surveys, and expert synthesis into a coherent user-centered design pipeline.The pipeline captures naturalistic intent, elicits goals and satisfaction, and organizes findings into actionable requirements.
  • Critical user journeys: The analysis identifies four representative critical user-journey categories: general health knowledge, personal data insights, wellness advice, and personal medical symptoms.Together they span factual understanding, interpretation of personal measurements, actionable guidance, and symptom-related questions.
  • General health knowledge: General health-knowledge queries seek facts, comparisons, treatment or behavior trade-offs, and updates across medical, fitness, and wellness topics.Examples include questions about infectiousness, HIIT pros and cons, and differences between HRV and resting heart rate.
  • Personal data insights: Personal-data queries ask users’ wearable or medical-record data to be interpreted through patterns, relationships, guideline comparisons, and meaningful changes.Examples include relating sleep scores to resting heart rate, assessing step impacts, and identifying unusual HRV events.
  • Wellness advice: Wellness-advice queries request general or data-specific recommendations, goal planning, obstacle handling, and progress tracking.Examples cover improving sleep scores, adapting workouts to weather, and identifying influences on blood pressure.
  • Personal medical symptoms: Symptom-related journeys may require back-and-forth conversation to gather additional information about personal experiences and potential causes.The category includes questions about symptoms such as an itchy throat, swallowing difficulty, or possible high blood pressure.
  • Scope and boundaries: The categories are not mutually exclusive for individual queries or conversations and do not exhaustively represent all user needs.The authors exclude less health-related topics such as privacy, safety, and social relationships.

3. Personal Health Agent and Evaluation Framework

The framework combines three specialized agents with a central orchestrator to address diverse health queries, then evaluates both individual capabilities and integrated performance. Real-world validation uses multimodal wearable and health-record data, while human evaluations draw on clinically identified participant profiles.

  • Personal Health Agent: The DS, DE, and HC agents provide complementary capabilities for personal data analysis, contextual health reasoning, and personalized coaching.The system is designed around specialized components rather than a single undifferentiated capability set.
  • Personal Health Agent: The PHA orchestrator parses each query, decomposes it into subtasks, routes them to appropriate agents, and synthesizes their responses.This collaboration supports queries that require data analysis, domain knowledge, and coaching in one interaction.
  • Evaluation Framework: The studies use single- and multi-turn interactions, single- and multi-modal data, automatic and human evaluations, and perspectives from health experts and end-users.This mixed-methods design is intended to provide a broad assessment of agent performance.
  • Evaluation Framework: The evaluation framework assesses DS analysis planning and executable code, DE factual and contextual reasoning, HC coaching effectiveness and fidelity, and holistic PHA conversations.The framework covers both component-level competencies and the integrated system’s conversational performance.
  • Experimental Setup: Gemini 2 Flash and Pro serve as base models because the work focuses on constructing the agentic system rather than training a new LLM.The authors state that this choice supports evaluation without training-data contamination or use of real participant health data for training.
  • Real-World Dataset for Evaluation: The WEAR-ME study contributes multimodal Fitbit, blood-test, and questionnaire data, while human evaluations use 50 personas from 10 common health profiles.The evaluation cohort randomly selected five individuals for each clinician-identified profile.

4. Data Science Agent

The Data Science Agent is designed to translate ambiguous personal-health questions into rigorous, executable analyses and valid numerical insights. Its structured planning and code-generation workflow substantially outperformed the Base Gemini model, especially in grounding analyses in data constraints and producing working code.

  • Motivation: LLM-based agents must interpret, decompose, and operationalize open-ended personal-health inquiries into statistically sound analyses.This is especially important because flawed analytical decisions can produce misleading or harmful conclusions.
  • Capabilities: The DS Agent addresses numerical queries by aligning statistical outcomes with user intent, context, timeframes, assumptions, and data characteristics.Its capability scope covers wearable streams and medical records, including the surrounding context needed to interpret results.
  • Capabilities: The agent decomposes broad queries into structured plans by identifying conceptual variables, specifying transformations, and checking for missing or insufficient data.Transformations may include filtering, cleaning, joining, and aggregating data across user and population-level tables.
  • Architecture: The first stage converts an open-ended query into a precise analysis plan grounded in schemas, column descriptions, semantic tags, and summary statistics.The plan explicitly specifies variables, transformations, and the selected statistical model before implementation.
  • Evaluation: 75.6±1.4% versus 53.7±1.8% was the DS Agent’s average analysis-plan quality compared with Base Gemini, with the largest gain in data-availability checking.Data Availability Check rose from 5.3±1.6% to 66.5±3.7%, while timeframe identification reached 96.0±1.9%.
  • Evaluation: 75.5±3.3% versus 58.4±3.7% was the first-attempt code pass rate for the DS Agent and Base Gemini, respectively, increasing to 79.0% after five trials.Data-handling errors also fell from 25.4±3.3% to 11.0±2.4%, indicating more robust pipeline construction.

5. Domain Expert Agent

The Domain Expert Agent is designed to synthesize authoritative medical knowledge with multimodal personal health data, producing personalized, evidence-based responses and summaries. Across diagnostic, user, and clinician evaluations, it generally outperformed the relevant base or comparison models.

  • Capabilities: The DE Agent addresses the need to personalize health-domain reasoning over contextual, wearable, and medical-record data rather than only retrieve general medical knowledge.It is designed to synthesize, contextualize, and validate information against authoritative sources and patient-specific data.
  • Capabilities: Its multi-step architecture combines user queries and personal health data with web, NCBI, DataCommons, and Python tools to produce auditable, evidence-based conclusions.The iterative process creates a transparent reasoning trace while grounding answers in authoritative sources and personal data.
  • Evaluation: 83.6% overall accuracy versus 81.8% for base Gemini showed statistically significant improvement across all questions.The comparison used McNemar’s Test, with χ2 = 9.506, p= 0.002, and Odds Ratio 1.667.
  • Evaluation: The DE Agent achieved 46.1% top-1 diagnostic accuracy versus 41.4% for the DDx Agent, a performance advantage Δ=4.7%.It also reached 75.6% top-5 accuracy and 84.5% top-10 accuracy, with significant overall diagnostic improvement.
  • Evaluation: End-users rated DE Agent responses as more trustworthy and personalized, including 96.9±0.8% Trustworthiness and 71.9±1.5% preference for Personalization-and-Contextualization.Relevance-to-Query ratings were similar between models, while Citation-Credibility favored the DE Agent in 76.9±1.5% of comparisons.
  • Evaluation: Clinicians preferred DE Agent multimodal summaries for clinical significance, cross-modal association, comprehensiveness, citation credibility, trustworthiness, and usefulness.The DE Agent received 93.3±1.9% of clinician preferences for Comprehensiveness and 82.4±3.0% for Trustworthiness.

6. Health Coach Agent

The Health Coach Agent is built for evolving goals, ambiguous context, and mixed-initiative dialogue, using modular coaching and conversation-flow components. End-users and experts generally preferred it over a baseline, though progress measurement remained weaker.

  • Motivation: Health coaching requires iterative clarification, balancing user-led exploration with proactive guidance, and timing recommendations to avoid premature advice or excessive probing.These challenges affect engagement, trust, frustration, and confidence in the agent.
  • Capabilities: Experts identified capabilities spanning goal identification, active listening, context clarification, user empowerment, concrete SMART recommendations, and feedback incorporation.The capabilities were derived from formative interviews with 11 health experts across sleep, fitness, and life coaching.
  • Architecture: The modular HC architecture combines a Personalized Coaching Module with Recommendation and Conclusion flow modules to adapt dialogue and time recommendations strategically.The coaching process is structured to identify goals, explore context and constraints, solicit feedback, and then provide recommendations.
  • End-user evaluation: End-users rated the HC Agent higher on conversation flow, motivational interviewing, and feedback incorporation than the baseline.Reported scores were 75.5 ± 3.3% versus 64.5 ± 3.4% for conversation flow, 65.8±3.2% versus 57.1±3.8% for motivational interviews, and 66.1 ± 2.9% versus 60.7 ± 3.8% for feedback incorporation.
  • Expert evaluation: Experts found advantages of Δ = 11.0 −12.8% across most expert themes and a 10.2 ± 1.2% relative advantage, exceeding the end-user advantage of 3.5 ± 2.5%.Experts also preferred the HC Agent on Working-Alliance and Goal-Alignment, while clarity was tied.
  • Expert evaluation: Progress measurement was the main exception: experts rated the baseline 50.3 ± 2.4% versus 46.7 ± 1.4% for the HC Agent.The authors relate this difference to the HC Agent allocating more interaction content to context gathering and goal alignment.

7. Personal Health Agent: A Multi-Agent Framework

The PHA combines three specialized agents through synergistic collaboration, dynamic task assignment, and iterative orchestration to address multifaceted personal-health needs. Across end-user and expert evaluations, it was preferred over single-agent and parallel multi-agent baselines, with experts reporting especially large advantages.

  • Framework design: The PHA assembles data-science, domain-expertise, and health-coaching agents into a cohesive system because multifaceted personal-health needs require synergistic collaboration.The modular design also supports explainability by tracing outputs to each agent’s function.
  • Framework design: The Orchestrator dynamically assigns main and supporting agents, decomposes queries into targeted sub-queries, and adapts collaboration to each user need.For sleep-improvement queries, the health coach can lead while the data-science agent first analyzes sleep data.
  • End-user evaluation: The PHA was ranked best for overall preference 48.7% of the time and for response quality 49.1%, exceeding parallel multi-agent results of 24.8% and 24.6%.Both comparisons were statistically significant (p_s<0.001).
  • End-user evaluation: PHA achieved 70.3 ± 0.7% on Personal-Data-Analysis, compared with 54.9 ± 0.8% for parallel and 59.7 ± 0.8% for single-agent systems.The advantage was statistically significant (p<0.001).
  • Expert evaluation: Experts rated PHA response quality at 84.8±0.9%, versus 70.8±0.9% for parallel multi-agent and 65.2±0.8% for single-agent systems.PHA also showed significant advantages across data science, domain knowledge, and coaching metrics.
  • Expert evaluation: Experts ranked PHA best for quality and effectiveness in 80.0±2.5% of cases, while end-users rated single-agent and parallel baselines similarly.The difference suggests users valued coherent narratives, whereas experts recognized specialized depth despite imperfect synthesis.

8. Discussion

The discussion finds that PHA’s modular, multi-agent design improves personalized health-query performance while exposing limitations in reasoning, adaptation, progress tracking, safety, and long-term evaluation.

  • Data Science Agent: 75.6% mean plan score versus 53.7% baseline shows strong DS Agent proficiency, but statistical-distribution reasoning and executable code generation remain constrained.Code generation reached 79.0% but often required iterative correction, producing 20–50 second latency.
  • Domain Expert Agent: 86.6% and 64.8% gains in clinician-rated comprehensiveness and trustworthiness, respectively, accompanied the DE Agent’s factuality and citation tools.The agent also achieved a 43.8% increase in user-rated Contextualization-and-Personalization win rate.
  • Health Coach Agent: The HC Agent improved complex dialogue, motivational interviewing, and feedback adaptation, but stylistic interaction cannot compensate for missing personalized, actionable, context-sensitive guidance.Users may view coaching as inauthentic or ineffective when core competencies are absent.
  • Limitations and Future Directions: PHA’s current limitations include below-baseline progress measurement, insufficient adaptation to dynamic psychological states, and risks from cooperative-agent misalignment.Future collaboration designs may require dynamic agent selection and robust adjudication mechanisms.
  • Multi-Agent Framework: PHA’s modular architecture supports independent evaluation and specialized development, while its multidisciplinary synthesis improved complex-query performance over baselines.Advantages averaged 4.6% for end-users and 17.1% for experts.
  • Safety and Scope: PHA is not designed to provide definitive diagnoses or replace qualified healthcare professionals, and real-world application requires clinical safety and regulatory analysis.The framework is presented as a potential tool for helping users interpret experiences while seeking professional care.

9. Conclusion

The paper presents PHA as a multi-agent framework for contextualized, personalized, and evidence-based health support, evaluated across individual agents and integrated interactions.

  • PHA orchestrates Data Science, Domain Expert, and Health Coach agents to deliver contextualized, personalized, and evidence-based responses.The framework addresses complex personal health support through complementary specialist roles.
  • The system was evaluated across 10 benchmark tasks with more than 1,100 hours of human evaluation involving end-users and health experts.
  • The authors report advantages for both individual agents and the integrated multi-agent framework over baseline solutions.

Supplementary Material

Supplementary materials map personal health queries to specialized agents and workflows, illustrating how data analysis, interpretation, advice, and goal support are assigned or combined.

  • Table S1 organizes four critical user journeys and 26 common query types with assigned Data Science, Domain Expert, and Health Coach workflows.
  • Data Science handles numerical summaries, comparisons, relationships, changes, and computations involving personal health data.
  • Domain Expert interprets personal data, answers health questions, explains symptoms, and provides domain-specific knowledge when complex computation is unnecessary.
  • Health Coach provides general or personalized wellness advice, sets goals, makes plans, tracks progress, and adjusts support to changing needs.
  • Queries such as improving deep sleep may require collaboration among Data Science, Domain Expert, and Health Coach agents.

A.2. User-Centered Survey Details

The supplementary survey details describe an end-user study of Ask Health Fitbit Labs covering satisfaction, usefulness, trust, credibility, goals, and perceived benefits or shortcomings.

  • The survey was delivered to end-users who interacted with Ask Health Fitbit Labs, with N=555 participants.
  • Survey items assessed satisfaction with answers and suggested questions, including open-ended explanations of satisfaction or dissatisfaction.
  • Participants rated whether the system improved understanding of health and fitness and selected desired benefits such as comparisons, correlations, and personalized recommendations.
  • The survey measured perceived technological advancement, uniqueness, usefulness, trust, and credibility using structured rating scales.
  • The study’s data categories included demographics, anthropometrics, socioeconomic status, health-related characteristics, behaviors, wearable metrics, and blood biomarkers.

B.2.1. User Persona Examples for Evaluation Studies

The supplementary evaluation materials define persona examples with health records, wearable data, user stories, goals, and analysis-plan assessment procedures.

  • Persona examples combine demographics, medical conditions, blood-test values, wearable records, user stories, and health goals.
  • One persona sought to reduce BMI below 30, return to 10,000 daily steps, and avoid hospital readmission.
  • The supplied wearable schema includes daily steps, sleep duration and stages, resting heart rate, heart-rate variability, stress, active-zone minutes, and activity records.
  • Activity records include timing, activity type, distance, duration, elevation gain, average heart rate, and calories.
  • After removing 7 of 33 rubric items with low agreement, the final analysis-plan rubric used 26 items.
  • The analysis-plan performance metric was computed as the ratio of non-deducted scores to total scores, with up to 16 points deducted per approach.

C.5. DS Agent Task 1 Auto-Rater Development and Reliability

The study developed an LLM-based autorater to scale evaluation of DS Agent analysis plans and assessed its agreement with human experts. Agreement was high, with moderate reliability for inferred point deductions.

  • Autorater development: The autorater receives the query, data schema, and analysis plan, then evaluates rubric items independently using Gemini 2.0 Flash.The overall alignment item additionally uses the autorater’s prior rubric responses as context.
  • Reliability: 82.4% agreement with human experts slightly exceeded human-human agreement, while agreement with the human majority reached 86.9%.These results are reported for the overall test set.
  • Reliability: ICC(3,1) was 0.838 for analysis-plan quality, indicating moderately reliable consistency between autorater-inferred and human-assigned point deductions.The paper considers this reliability suitable for scaling analysis-plan assessment.
  • Evaluation design: Seven of 33 rubric items were excluded from human-human agreement calculations because they rarely applied and introduced noise.The excluded items focused on hard-coded values and distributional considerations in summary statistics.

C.6.2. Task 2: Code Generation and Execution

This task evaluates code generation and execution errors for the DS Agent. Data handling and total errors were substantially higher in one compared condition, while other error categories showed smaller, nonsignificant differences.

  • Error rates: 25.4±3.3% data handling errors (44/173) versus 11.0±2.4% (19/173) showed a significant difference (χ2=12.3, p<0.001).Other error categories were only counted when no data handling error occurred.
  • Error categories: 12.4±3.0% versus 10.4±2.4% general programming errors showed no significant difference (Z=0.53, p=0.594).The compared counts were 16/129 and 16/154, respectively.
  • Error categories: 7.0±2.2% versus 3.9±1.5% calculation/logic errors showed no significant difference (Z=1.15, p=0.249).The compared counts were 9/129 and 6/154, respectively.
  • Error categories: 2.3±1.3% versus 0.6±0.7% output content errors showed no significant difference (Z=1.19, p=0.234).The compared counts were 3/129 and 1/154, respectively.
  • Error rates: 41.6±3.7% total errors (72/173) versus 24.3±3.3% (42/173) showed a significant difference (χ2=26.3, p<0.001).McNemar’s tests were used for paired data-handling and total-error comparisons.

D.4.1. Task 1: Answering Medically-Relevant Multiple Choice Questions

The DE Agent was evaluated on more than 2,000 expert-level health MCQs spanning Cardiology, Endocrinology, Sleep, and Activity. It achieved higher accuracy than base Gemini across all four domains, with the largest improvement on Activity questions.

  • Overall performance: The DE Agent consistently achieved higher accuracy than base Gemini across Cardiology, Endocrinology, Sleep, and Activity MCQ benchmarks.The evaluation used four curated datasets totaling more than 2,000 test questions.
  • Domain results: 89.7% (±1.3%) versus 86.2% (±1.5%) was the DE Agent’s largest reported accuracy advantage on Fitness examination MCQs (p<0.001).The paper identifies this as the largest improvement among the reported examinations.
  • Domain results: 88.0% (±1.3%) versus 86.4% (±1.4%) gave the DE Agent a 1.58% absolute improvement across 634 Sleep examination questions.The Sleep comparison is reported directly against base Gemini.
  • Domain results: 82.7% (±1.9%) versus 80.7% (±2.0%) represented a 2.01% improvement for the DE Agent on Endocrinology questions.The reported accuracies are for the DE Agent and base Gemini, respectively.
  • Domain results: 73.7% (±1.8%) versus 73.2% (±1.9%) were the reported DE Agent and base Gemini accuracies on Cardiology questions.The passage reports the two accuracies without a stated significance result.

D.4.2. Task 2: Diagnostic Conversations

The diagnostic-conversation evaluation used self-reported health cases with ground-truth diagnoses and compared the DE Agent with a DDx Agent. The DE Agent placed the correct diagnosis higher across top-1, top-5, and top-10 rankings.

  • Evaluation setup: 2,000 de-identified, self-reported health cases were evaluated, each containing symptoms and a ground-truth diagnosis.The cases originated from users recruited across 39 reported health-condition categories; filtering details are described in the figure materials.
  • Case construction: The benchmark categories were based on common patient language rather than medical-literature terminology.The paper frames this grounding as relevant to practical symptom-checking evaluation.
  • Diagnostic ranking: 46.1% top-1 accuracy versus 41.4% gave the DE Agent a 4.7-percentage-point gain over the DDx Agent.Correct diagnoses were counted when included among the agent’s top k predictions.
  • Diagnostic ranking: 75.6% top-5 accuracy versus 71.4% gave the DE Agent a 3.8% gain over the DDx Agent.The comparison used the rank of the correct diagnosis within the top 10 predictions.
  • Diagnostic ranking: 84.5% top-10 accuracy versus 81.1% gave the DE Agent a 3.1% gain over the DDx Agent.Cases outside the top 10 received a penalty rank of 11 in the ranking analysis.

E.4.2. Task 2: Validation of Health Coach Agent Design with Expert Evaluation

Expert evaluation of the Health Coach Agent used side-by-side comparisons with three experts, measuring win rates and applying bootstrap-based uncertainty estimation. The broader PHA workflow assigns agents and specifies collaboration procedures for user queries.

  • Expert evaluation: N=3 experts evaluated the HC Agent side-by-side, with performance measured as win rate and good inter-rater reliability (Fleiss’ κ=0.640).Bootstrap sampling with replacement estimated standard errors, and a binomial GLMM tested differences.
  • Agent orchestration: The PHA orchestrator assigns DS, DE, and HC agents according to the user’s query and determines their collaboration workflow.The prompt identifies a main agent, supporting agents, and a collaboration workflow for each request.
  • Agent orchestration: The orchestration prompt checks unsupported corner cases, matches supported questions to predefined categories, and falls back to basic Gemini when no category applies.A further rule adds the DS Agent when a query is potentially relevant to personal data.
  • Agent orchestration: For time-series questions requiring external health knowledge, the DS Agent computes requested results before the DE Agent interprets them.This workflow applies when users ask why data changed, compare data with others or guidelines, or relate data across sources.

F.4.1. Task 1: Multi-agent Conversation Experience for End-Users

End-user and expert evaluations compared single-agent, parallel multi-agent, and PHA systems using rubric scores and ranking outcomes. The reported tables include results across individual agent capabilities and combined tasks.

  • End-user evaluation: Overall Quality (DS + DE + HC) was 82.3±0.3 %, 82.7±0.3 %, and 85.1±0.4 % across the three evaluated systems.The table also reports combined-task metrics for health data interpretation, individualized suggestions, and grounded suggestions.
  • Expert evaluation: PHA rubric evaluations by experts used N=5 and reported moderate-to-good inter-rater reliability, including ICC(3,1)=0.690 for PHA.The comparison included single-agent PHIA and a parallel multi-agent system.
  • End-user evaluation: Asks Clarifying Questions (HC) was 38.6±1.1 %, 49.4±1.2 %, and 77.6±1.2 % across the three evaluated systems.The same table reports Understanding & Engagement (HC) values of 55.4±0.9 %, 66.4±1.0 %, and 86.1±0.8 %.
  • Expert evaluation: Expert side-by-side comparisons reported an 81.5% agreement rate, using ranking rates and bootstrap-estimated standard errors.Friedman tests were followed by post-hoc Wilcoxon rank-sum tests with Holm-Bonferroni correction.
Loading 2508.20148v2…