PaperlayerView exact revision on arXiv

Source-linked AI summary

The Anatomy of a Personal Health Agent

A. Ali Heydari, Ken Gu, Vidya Srinivas, Hong Yu, Zhihan Zhang, Yuwei Zhang, Akshay Paruchuri, Qian He, Hamid Palangi, Nova Hammerquist, Ahmed A. Metwally, Brent Winslow, Yubin Kim, Kumar Ayush, Yuzhe Yang, Girish Narayanswamy, Maxwell A. Xu, Jake Garrison, Amy Armento Lee, Jenny Vafeiadou, Ben Graef, Isaac R. Galatzer-Levy, Erik Schenck, Andrew Barakat, Javier Perez, Jacqueline Shreibati, John Hernandez, Anthony Z. Faranesh, Javier L. Prieto, Connor Heneghan, Yun Liu, Jiening Zhan, Mark Malhotra, Shwetak Patel, Tim Althoff, Xin Liu, Daniel McDuff, Xuhai "Orson" Xu

arXiv:2508.20148v2cs.AIcs.HCcs.MA

TL;DR

It remains unclear whether LLM agents can handle open-ended, personalized health-data questions, and no systematic benchmark evaluates personal-health analysis planning. This paper presents PHA, which coordinates data science, health-domain, and coaching agents; across 10 benchmark tasks, the authors report advantages over baseline solutions.

  • Problem

    Whether LLM agents can interpret and operationalize open-ended, personalized health-data inquiries remains unclear, and personal-health analysis planning lacks a systematic benchmark despite the risks of flawed decisions.

    Source: §4 · p. 10

  • Method

    PHA coordinates data science, health-domain expert, and health coach sub-agents to address user queries spanning their respective capabilities.

    Source: §1 · p. 4, §9 · p. 38

  • Results

    Across 10 benchmark tasks, the authors report advantages over baseline solutions for the individual agents and the multi-agent framework.

    Source: §9 · p. 38

  • Takeaways & Limitations

    The work establishes a concrete foundation for future development of accessible personal health agents to help improve everyday health and well-being.

    Source: §1 · p. 4

    The evaluation does not assess PHA’s long-term impact on user behavior through multi-session interactions.

    Source: Limitations · p. 37

Abstract

from arXiv · show

Health is a fundamental pillar of human wellness, and the rapid advancements in large language models (LLMs) have driven the development of a new generation of health agents. However, the application of health agents to fulfill the diverse needs of individuals in daily non-clinical settings is underexplored. In this work, we aim to build a comprehensive personal health agent that is able to reason about multimodal data from everyday consumer wellness devices and common personal health records, and provide personalized health recommendations. To understand end-users' needs when interacting with such an assistant, we conducted an in-depth analysis of web search and health forum queries, alongside qualitative insights from users and health experts gathered through a user-centered design process. Based on these findings, we identified three major categories of consumer health needs, each of which is supported by a specialist sub-agent: (1) a data science agent that analyzes personal time-series wearable and health record data, (2) a health domain expert agent that integrates users' health and contextual data to generate accurate, personalized insights, and (3) a health coach agent that synthesizes data insights, guiding users using a specified psychological strategy and tracking users' progress. Furthermore, we propose and develop the Personal Health Agent (PHA), a multi-agent framework that enables dynamic, personalized interactions to address individual health needs. To evaluate each sub-agent and the multi-agent system, we conducted automated and human evaluations across 10 benchmark tasks, involving more than 7,000 annotations and 1,100 hours of effort from health experts and end-users. Our work represents the most comprehensive evaluation of a health agent to date and establishes a strong foundation towards the futuristic vision of a personal health agent accessible to everyone.

1. Introduction

The paper proposes a multi-agent personal health agent that combines analysis of wearable and medical-record data, contextual health interpretation, and personalized coaching to address diverse everyday health needs. It also describes a holistic evaluation of the specialist agents and integrated system.

1. Introduction

  • Prior work leaves gaps in numerical time-series reasoning, broad query coverage, domain knowledge, and interactive coaching, motivating a comprehensive agent for personalized health needs.Khasentino et al. (2025) examined a narrow set of fitness and sleep insights, while Merrill et al. (2024a) covered broader numerical queries but did not evaluate domain knowledge or expert-level interactive coaching.Source: §1 · p. 2
  • Everyday health queries span four categories: general health knowledge, personal data, wellness advice, and personal health symptoms.These categories were identified by synthesizing end-user online queries and discussion, survey results, and an expert design workshop.Source: §1 · p. 2
  • The DS Agent analyzes wearable and population time-series data, the DE Agent interprets personal records and wearable data using health knowledge, and the HC Agent supports behavior change through evidence-based psychological strategies.The coach helps users set goals, identify barriers, and develop personalized plans.Source: §1 · pp. 2, 4
  • The Personal Health Agent combines an orchestrator with three specialized agents to coordinate data analysis, health expertise, and coaching across user queries.The framework is designed to provide holistic, actionable insights and is not intended to supplant human health providers.Source: §1 · p. 4
  • The evaluation covers 10 benchmark tasks and more than 7,000 human annotations across 1,120 hours of effort: 559 expert hours and 561 end-user hours.It assesses individual agent capabilities and the integrated PHA through studies with end-users and health experts.Source: §1 · p. 4

2. A User-Centered Design of Personal Health Agent Requirements

The paper derives personal health agent requirements through a user-centered pipeline that combines real-world queries, surveys, and expert synthesis, then organizes users’ needs into four critical journey categories.

2. A User-Centered Design of Personal Health Agent Requirements

  • The design process moved from broad consumer-query intent to structured survey insights and expert workshops that distilled findings into actionable requirements.The three complementary layers were real-world user queries, targeted survey responses, and expert synthesis.Source: §2 · pp. 4–5

2.1. User-Centric Data

  • The evidence base included 1,370 queries, surveys from 555 participants, and a 14-expert workshop that used thematic analysis to develop a hierarchical user-journey codebook.Surveys covered users’ goals and sentiments, experience and satisfaction, engagement and retention, and prior experience with wearables, AI, and conversational agents.Source: §2.1 · p. 5

2.2. Major Categories of Use-Cases with Personal Health Agent

  • The analysis identified four overlapping journeys: general health knowledge, personal-data insights, actionable wellness advice, and questions about personal symptoms.The categories are not mutually exclusive, and the framework excludes less health-related needs such as privacy, safety, and social relationships.Source: §2.2 · pp. 5–6

3. Personal Health Agent and Evaluation Framework

The paper proposes a personal health agent that combines specialized data analysis, health expertise, and coaching, and evaluates its components and overall system through a multifaceted framework.

3.1. User-Centered Agent Design

  • The PHA brings together data analysis, health domain expertise, and behavior coaching as complementary sub-agents for diverse consumer health needs.The DS agent analyzes personal and population-level time-series data; the DE agent integrates wearable data, medical records, and context; and the HC agent supports open-ended wellness coaching, goals, and motivation.Source: §3 · p. 6, §3.1 · p. 7
  • A central orchestrator parses queries, routes subtasks to suitable agents, synthesizes and iteratively refines responses; the intended real-world role is to empower, not replace, clinicians.The paper frames its work on symptom assessment as research, while envisioning agents that provide contextual information and connect users with clinical resources.Source: §3.1 · pp. 6–7

3.2. Agent Evaluation Framework

  • The mixed-methods framework examines single- and multi-turn interactions, single- and multimodal data, automatic and human evaluations, and expert and end-user perspectives.The authors characterize it as the most comprehensive evaluation of a consumer AI health agent to date.Source: §3.2 · pp. 7–8
  • The evaluation framework tests DS planning and code execution, four DE competencies, HC coaching effectiveness and fidelity, and holistic PHA performance from user and expert perspectives.The DE competencies cover common health questions, diagnostic conversations, context-based personalization, and reasoning across multimodal health data.Source: §3.2 · p. 8

3.3. Choice of Base Language Model

  • Experiments use Gemini 2 Flash/Pro as the base model, while the agent and team designs are described as agnostic to the chosen LLM.The choice aimed to avoid training-data contamination and prevent participant health data from being logged for future model training.Source: §3.3 · p. 8

3.4. Real-World Dataset for Evaluation

  • The real-world evaluation uses WEAR-ME, a 1,165-participant metabolic health study with questionnaire, Fitbit, and blood-test data.For human evaluations, the study selected five participants at random from each of 10 common health profiles, yielding 50 personas; the study had IRB approval and participant e-consent.Source: §3.4 · pp. 8, 10, Figure 2 · p. 9

4. Data Science Agent

The Data Science Agent targets open-ended personal-health questions that require turning wearable and health-record data into valid, actionable numerical insights. Its two-stage design separates analysis planning from code execution, and its components are evaluated independently.

4.1. Data Science Agent Capabilities

  • The agent must translate ambiguous queries into an intent-aligned analysis plan, operationalize variables across data tables, address insufficient data, and compute accurate results with code.Plans should select appropriate transformations and statistical methods, account for context such as timeframe and assumptions, and avoid unsupported conclusions; one suitable interpretation may suffice in an interaction.Source: §4.1 · pp. 10–11

4.2. Data Science Agent Architecture

  • The modular architecture first generates a structured plan from the query and data summaries, then generates and executes code with iterative error-feedback refinement.Separating semantic reasoning about what to do from code generation about how to do it is intended to reduce factual and logical errors.Source: §4.2 · p. 11

4.3.1. Task 1: Analysis Plan Generation

  • 75.6±1.4% vs. 53.7±1.8% average analysis-plan score: the DS Agent significantly outperformed the Gemini baseline.It also improved Data Availability Check scores from 5.3 ± 1.6% to 66.5 ± 3.7%; Statistical Tests showed no significant difference (p=0.630).Source: Figure 3 · p. 13, §4.3.1 · p. 14

4.3.2. Task 2: Code Generation and Execution

  • 75.5±3.3% vs. 58.4±3.7% first-attempt code pass rate: the DS Agent significantly outperformed Base Gemini, reaching 79.0% after five trials.Data handling errors fell from 25.4±3.3% to 11.0±2.4%, while differences in other error categories were not statistically significant.Source: Figure 3 · p. 13, §4.3.2 · p. 15

4.3.1. Task 1: Analysis Plan Generation

  • A 141-pair expert-rated benchmark and a Gemini 2.0 Flash autorater with ICC(3,1)=0.838 enabled scalable analysis-plan evaluation.Ten data scientists supplied 354 approach annotations and 6,157 rubric-item annotations, with 80.2% average agreement.Source: §4.3.1 · p. 14

4.3.2. Task 2: Code Generation and Execution

  • Code-generation evaluation used 25 expert-authored query-approach test suites containing 173 unit tests to assess functional correctness and robustness.The tests covered limited data, complex transformations, statistical methods, and answer accuracy; models could retry up to five times after execution errors.Source: §4.3.2 · pp. 14–15

5. Domain Expert Agent

The Domain Expert Agent (DE) addresses the risk that general-purpose LLMs may give misleading health information by grounding and contextualizing medical knowledge with authoritative sources and users’ health data. It is designed to turn complex personal health information into accurate, personalized insights, and is evaluated across four capabilities.

5.1. Domain Expert Agent Capabilities

  • The four evaluated capabilities are reliable health information, differential diagnosis, personalization to a user’s profile and goals, and reasoning across wearable data and medical records.These capabilities were investigated in a controlled academic context and do not establish the readiness of real-world products requiring regulatory evaluation.Source: §5.1 · pp. 16, 18

5.3.4. Task 4: Reasoning over Multi-Modal Personal Health Data

  • In clinician evaluations (N=5), DE Agent summaries outperformed the base Gemini model on Clinical-Significance (96.4±0.6% vs. 73.8±0.6%) and were preferred for Comprehensiveness in 93.3±1.9% of comparisons.Source: Figure 4 · p. 17, §5.3.4 · p. 21

5.3.3. Task 3: Contextualized and Personalized Responses to Health Queries

  • 96.9±0.8% vs. 38.7±3.3% Trustworthiness; end-users preferred the DE Agent for Personalization-and-Contextualization in 71.9±1.5% and Credibility-of-Citations in 76.9±1.5% of comparisons.The models performed similarly on Relevance-to-Query (97.6±0.7% vs. 98.1±0.6%).Source: Figure 4 · p. 17, §5.3.3 · pp. 20–21

5.2. Domain Expert Agent Architecture

  • The DE Agent uses a multi-step Reason-Investigate-Examine cycle, selecting tools such as Web Search, NCBI, and DataCommons to iteratively investigate a query and examine results.Its auditable reasoning trace combines authoritative-source facts with the user’s personal health data to produce evidence-based answers.Source: §5.2 · p. 18

5.3. Domain Expert Agent Evaluation

  • The DE Agent’s four capabilities were evaluated through four independent tasks.The tasks examined factual health knowledge, diagnostic conversations, contextualized responses, and multimodal health summaries.Source: §5.3 · p. 18

5.3.1. Task 1: Ground Truth of Knowledge Taxonomy and Factuality

  • 83.6% overall accuracy vs. Gemini’s 81.8%, a statistically significant improvement across more than 2,000 expert-level questions (p=0.002).The four question sets covered endocrinology, cardiology, fitness, and sleep medicine.Source: §5.3.1 · pp. 18–19

5.3.2. Task 2: Diagnostic Conversations

  • 46.1% vs. 41.4% top-1 accuracy; DE Agent top-5 accuracy was 75.6% (Δ=3.8%) and top-10 accuracy was 84.5% (Δ=3.1%), outperforming the DDx Agent.The comparison used top-10 diagnoses generated from consumer self-reports in a retrospective study of 2,000 participants.Source: §5.3.2 · pp. 19–20

6. Health Coach Agent

The Health Coach Agent addresses personalized coaching’s multi-turn challenges through expert-informed modular dialogue; end-users and coaches rated it more favorably than a baseline across most evaluated dimensions.

6. Health Coach Agent

  • Coaching must handle evolving open-ended goals, incomplete context, mixed-initiative dialogue, and conversational diversions while balancing clarification against timely actionable recommendations.Premature advice can seem dismissive and undermine trust, while excessive probing can frustrate users and reduce confidence.Source: §6 · p. 22

6.1. Health Coach Agent Capabilities

  • Expert interviews informed six HC Agent capabilities: identify goals, listen actively, clarify context, empower users, give SMART recommendations, and incorporate feedback.Source: §6.1 · p. 23

6.2. Health Coach Agent Architecture

  • The modular architecture combines personalized coaching with recommendation and conclusion modules to adapt dialogue and time advice appropriately.The personalized module uses motivational-interviewing-inspired behaviors, guiding the agent to identify goals, explore context and constraints, and solicit feedback before recommending; the flow modules manage when to advise and conclude.Source: §6.2 · pp. 23, 25

6.3.2. Task 2: Validation of Health Coach Agent Design with Expert Evaluation

  • Experts rated the HC Agent above baseline across almost all dimensions (expert-theme Δ = 11.0 −12.8%; AI-coaching Δ = 7.1−15.7%), with a 10.2 ± 1.2% relative advantage.Experts also preferred it on motivational-interview skills (59.1% win rate), recommendation quality (53.8%), credibility (66.7%), and themes HC1–HC6 (64.5-72.0%). The agents tied on AI-coaching clarity.Source: Figure 5 · p. 24, §6.3.2 · p. 26

6.3.1. Task 1: Health Coach Agent Capability from End-Users’ Perspectives

  • Users preferred the HC Agent in five of six themes, including significant active-listening preference (71.0% win rate, p= 0.025), and it produced more natural endings than baseline (83.9% vs. 74.2%).The baseline scored higher on user-rated progress measurement (61.3 ± 4.5% vs. 48.7 ± 3.9%).Source: §6.3.1 · p. 26

6.3.2. Task 2: Validation of Health Coach Agent Design with Expert Evaluation

  • Experts rated baseline higher on progress measurement (50.3 ± 2.4% vs. 46.7 ± 1.4%), while user and expert rankings diverged on empowerment (48.4% vs. 69.9% HC Agent win rates).The authors suggest the HC Agent’s limited-interaction design prioritizes context gathering and goal alignment over progress tracking.Source: §6.3.2 · p. 27

7. Personal Health Agent: A Multi-Agent Framework

The Personal Health Agent (PHA) coordinates three specialist agents in an iterative, memory-supported workflow to address diverse personal health needs. End-user and health-expert evaluations favored PHA over single-agent and parallel multi-agent baselines.

7.1. Multi-Agent Design Principles

  • PHA’s four design principles call for comprehensive user needs, adaptive support, low user burden, and simplicity, guided by modular cognition and complementary human-team expertise.The authors argue that assigning distinct cognitive roles supports holistic responses and makes decisions more explainable by tracing outputs to agent functions.Source: §7.1 · pp. 27–28

7.2. Multi-Agent Collaboration Framework

  • A Gemini-based Orchestrator identifies user needs, dynamically assigns a main agent and supporting agents, supports iterative reflection before response, and updates persistent memory after each turn.Memory retains insights, user goals, barriers, and preferences to support coherence and personalization across interactions.Source: §7.2 · pp. 28–29

7.3.2. Task 2: Multi-agent Conversation Experience for Experts

  • Experts rated PHA’s overall response quality at 84.8±0.9%, versus 70.8±0.9% for parallel multi-agent and 65.2±0.8% for single-agent.Experts ranked PHA best for quality and effectiveness in 80.0±2.5% of cases; the parallel system received 15.6±2.3% and 15.2±2.3%, and the single-agent 4.4±1.3% and 4.8±1.4%, respectively.Source: Figure 6 · p. 30, §7.3.2 · p. 33

7.3.1. Task 1: Multi-agent Conversation Experience for End-Users

  • Users ranked PHA best overall conversation in 48.7% and response quality in 49.1%, versus parallel 24.8% and 24.6% and single-agent 26.5% and 26.3% (ps<0.001).PHA’s iterative reflection and memory support a coherent, evolving conversation; evaluators found the parallel baseline could be confusing or disjointed without organic synthesis.Source: Figure 6 · p. 30, §7.3.1 · pp. 31–32

7.3. Personal Health Agent Evaluation

  • The evaluation used 150 multi-turn conversation scripts based on 50 user personas; 20 end-user evaluators rated conversations, and five health experts reviewed all scripts.The end-user evaluation produced 3,000 individual ratings and 1,000 comparative rankings; the expert evaluation produced 750 ratings and 250 rankings.Source: §7.3.1 · p. 31, §7.3.2 · p. 33

7.3.1. Task 1: Multi-agent Conversation Experience for End-Users

  • PHA scored 70.3 ± 0.7% on Personal-Data-Analysis versus 54.9 ± 0.8% parallel and 59.7 ± 0.8% single-agent, and 76.5 ± 0.6% on Data-Results-Interpretation versus 64.8 ± 0.8% and 69.1 ± 0.8%.The authors attribute the advantage to connecting quantitative data analysis with qualitative health interpretation through iterative reflection.Source: §7.3.1 · p. 32
  • PHA used 6.5 LLM calls per round and responded in 205.3 ± 15.0s, compared with 7.0 calls and 244.4 ± 11.4s for parallel multi-agent and 3.0 calls and 36.4 ± 1.6s for single-agent.Despite its reflection stages, PHA’s dynamic assignment required fewer calls and faster responses than the parallel baseline.Source: §7.3.1 · p. 32

8. Discussion

The discussion reports strengths and limitations across the specialist agents and PHA, and outlines future work on personalization, safety, and deployment. It emphasizes that PHA can integrate personal health data and clinical knowledge, but does not replace professional medical judgment.

Which Analyses Can the Data Science Agent Conduct?

  • The DS Agent scored 75.6% on plan auto-rating, compared with a 53.7% baseline, and operationalized ambiguous queries into statistical analysis plans.Plans often overlooked data distributions; code generation reached 79.0% but could require repeated correction and take 20-50 seconds, particularly for complex table joins.Source: Which Analyses Can the Data Science… · p. 34

Which Questions Can the Domain Expert Agent Accurately Answer?

  • The DE Agent improved comprehensiveness and trustworthiness by 86.6% and 64.8% versus baseline, respectively, and increased contextualization-and-personalization win rate by 43.8%.Limitations include incomplete reasoning chains in 8.5% of interactions, conflicting web sources, and risks of biased personalization; the authors recommend better source selection and diverse-persona audits.Source: Which Questions Can the Domain Expert… · pp. 34–35

What Are the Important Characteristics of an Health Coach Agent?

  • The HC Agent structured conversations, used motivational interviewing, and adapted to feedback, but coaching style helped only when grounded in personalized, actionable understanding of users’ goals and context.It scored below baseline on progress measurement; a PHA prompt update improved this capability, though further modular design may be needed.Source: What Are the Important Characteristics… · p. 35

What Are the Benefits of the Personal Health Agent? How to Further Improve?

  • End-users rated PHA’s advantage over baselines at 1.6%-15.4% (average 4.6%), while experts rated it at 5.7%-39.0% (average 17.1%).The modular framework supports independent evaluation and concurrent development of specialist agents while synthesizing clinical knowledge with personal health data for complex queries.Source: What Are the Benefits of the Personal… · pp. 35–36
  • Future PHA designs could dynamically adapt agent collaboration and conversational style, while requiring transparent conflict resolution and clinical safety and regulatory analysis before real-world use.The discussion contrasts cooperative agents, which risk inefficiencies and amplified failures, with competitive agents that need robust adjudication; adaptive interaction also requires modeling users’ changing psychological states.Source: What Are the Benefits of the Personal… · p. 36

Which User Journeys are Best Served?

  • PHA performs especially well in personal health data interpretation (CUJ2) and wellness advice (CUJ3), while symptom assessment (CUJ4) remains challenging and is not a substitute for diagnosis by clinicians.Future systems could help users log symptoms, identify patterns, prepare questions, and navigate care, with safeguards against premature or inaccurate diagnostic suggestions.Source: Which User Journeys are Best Served? · pp. 36–37

Risk and Ethical Considerations for Future Safeguards

  • Responsible PHA deployment requires safeguards for algorithmic bias, sensitive-data security, user over-reliance, and compliance with privacy and AI regulations.The authors call for bias audits and fairness mitigation, sandboxed code execution and validation, persistent disclaimers and high-risk query referral, and regulatory assessment.Source: Risk and Ethical Considerations for… · p. 37

Limitations

  • PHA’s evaluation has not yet assessed long-term behavioral impact through multi-session interactions, relies heavily on autoraters, and its three-agent-plus-orchestration implementation has substantial computational costs that present a barrier to scalability and widespread adoption.Future work will optimize the agent architecture and inference methods, and may leverage specialized hardware to reduce latency and cost.Source: Limitations · pp. 37–38

9. Conclusion

PHA decomposes personal health assistance into three specialist sub-agents and orchestrates them to provide contextualized, personalized, evidence-based responses. Evaluation across 10 benchmark tasks and more than 1,100 human evaluations showed advantages over baseline solutions.

9. Conclusion

  • PHA combines Data Science, Domain Expert, and Health Coach sub-agents, each designed to excel in its respective role.The framework orchestrates them to deliver contextualized, personalized, and evidence-based responses.Source: §9 · p. 38
  • PHA and its individual agents showed advantages over baseline solutions across 10 benchmark tasks and more than 1,100 human evaluations involving end-users and health experts.Source: §9 · p. 38
  • The work lays the foundation for a future personal health agent that supports human health expertise and is accessible to everyone.Source: §9 · p. 38

Supplementary Material · A. User-Centered Design & User Journey Details · A.1. User Journeys Breakdown with Agent Assignment

The user-centered design breakdown maps four critical user journeys and 26 common query types to workflows involving the Data Science (DS), Domain Expert (DE), and Health Coach (HC) agents.

A.1. User Journeys Breakdown with Agent Assignment

  • Table S1 organizes four critical user journeys and 26 common query types, with examples and workflows assigning the three agents to each user purpose.Researchers and health experts designed the workflows through the user-centered design process.Source: Table S1 · p. 47
  • For time-series questions, DS computes summaries, comparisons, and meaningful changes, such as longest sleep duration or average heart rate during an event.Example queries also ask to review HRV for unusual occurrences and summarize activity records over a training period.Source: §A.1 · p. 47
  • For health-record interpretation and relationships between data, DE interprets personal information, while DS computes when comparisons or cross-source analysis are needed.Examples include understanding a blood report, comparing exercise across sources, and exploring whether exercise relates to sleep or blood values.Source: §A.1 · p. 48
  • For general wellness advice, HC offers guidance; when advice depends on personal data, DS analyzes it and HC turns the results into personalized recommendations.The examples include tips to reduce smartphone use and improve deep sleep.Source: §A.1 · p. 48
  • For goal-setting and progress, HC tailors plans to users’ preferences, with DS analysis and DE interpretation included when appropriate.Example requests include setting a training plan, adjusting a workout to indoor conditions, and checking progress toward a marathon target.Source: §A.1 · p. 49
  • For personal symptoms, DE assesses the information provided and gives advice, with DS computation used when appropriate.Examples ask what might be causing an itchy throat with trouble swallowing and how to tell whether one has high blood pressure.Source: §A.1 · pp. 48–49

A.2. User-Centered Survey Details … B.2. User Profiles for Evaluation Studies

The supplementary sections describe a 555-user survey, the breadth of health data assembled for evaluation, and ten contextualized user profiles spanning wellness goals and chronic conditions.

A.2. User-Centered Survey Details

  • A 555-user Ask Health Fitbit Labs survey assessed goals and user experience, including answer quality, health understanding, trust, and information credibility.The survey also asked users what they hoped to achieve with Insights Explorer.Source: Table S2 · p. 50, §A.2 · pp. 51–53

B.1.1. Demographics; B.1.2. Anthropometrics; B.1.3. Socioeconomic Status

  • WEAR-ME records participant demographics, objectively measured and self-reported body measurements, and socioeconomic context including income, employment, and education.Demographic attributes include birth year, gender, ethnicity, and marital status; body measurements include height, weight, and waist circumference.Source: §B.1.1 · p. 54, §B.1.2 · p. 54, §B.1.3 · p. 54

B.1.4. Female & Reproductive Health; B.1.5. Lifestyle & Behavior (Baseline); B.1.6. Lifestyle & Behavior (Follow-up)

  • The dataset captures female reproductive status plus baseline and follow-up lifestyle, behavioral, dietary, substance-use, sleep, activity, and self-rated health measures.Follow-up measures include aerobic activity, eating habits, and overall self-rated health.Source: §B.1.4 · p. 54, §B.1.5 · p. 54, §B.1.6 · p. 54

B.1.7. Digital Biomarkers (Wearables)

  • Wearable data include high-frequency activity, sleep, cardiovascular, respiratory, oxygenation, temperature, fitness, stress, and raw sensor measures.Examples include daily steps, sleep stages and scores, resting heart rate, heart rate variability, SpO2, VO2 max, stress scores, and PPG.Source: §B.1.7 · p. 54

B.1.8. Blood Biomarkers (Lab Tests)

  • Blood-test data span lipid, metabolic, liver, glycemic, inflammatory, blood-count, and hormonal biomarkers.Measures include cholesterol, HDL, triglycerides, creatinine, sodium, ALT, AST, HbA1c, insulin, CRP, blood-cell counts, and testosterone.Source: §B.1.8 · pp. 54–55

B.1.9. Self-Reported Medical History; B.1.10. Self-Reported Clinical Data

  • Self-reported clinical data cover physician-diagnosed chronic conditions and diagnosis dates, along with blood pressure, cholesterol, and current medications.Conditions include cardiovascular disease, diabetes, hypertension, and kidney disease.Source: §B.1.9 · p. 55, §B.1.10 · p. 55

B.2. User Profiles for Evaluation Studies

  • Ten profiles cover four non-chronic wellness scenarios and six prevalent chronic-condition scenarios, allowing comorbidities and quantitative health metrics in the latter group.The non-chronic scenarios include athletic performance, a specific wellness goal, a sedentary lifestyle, and recovery from an acute disruption.Source: §B.2 · p. 55
  • Clinicians created each profile’s contextual persona from a user story and a specific, actionable health and wellness goal.The user story incorporates lifestyle, circumstances, health behaviors, and plausible personal or family medical history.Source: §B.2 · p. 55

B.2.1. User Persona Examples for Evaluation Studies … C.4.1. Task 1 Rubrics: Analysis Plan Generation

This appendix section documents evaluation personas and the data schemas, prompts, tests, and rubrics used in PHA’s data-analysis workflow. It covers analysis planning and code generation, including rubric criteria for statistical choices, data assumptions, and alignment with user intent.

B.2.1. User Persona Examples for Evaluation Studies

  • One persona describes a 67-year-old man with COPD, hypertension, and diabetes seeking more activity and steadier, shorter sleep; another describes a woman recovering from HHS who wants to regain activity and avoid readmission.The first persona reports limited walking due to lung problems and occasional nightmares; the second reports reduced activity after hospitalization, depression, and low-impact exercise goals.Source: §B.2.1 · pp. 56, 58

C.1. Conversation Examples

  • The appendix includes an example conversation between a user persona and the PHA data-science agent.The example is identified as a conversation with the DS Agent.Source: Figure S3 · p. 60

C.2.1. Summary Data Table

  • The summary data table represents activity, sleep, and personal health records by day, with measures including steps, sleep, resting heart rate, heart-rate variability, active-zone minutes, and stress management.The schema also includes sleep timing and stages, awake time, and fat-burn, cardio, and peak active-zone minutes.Source: §C.2.1 · pp. 61–62

C.2.2. Activities Data Table

  • The activities data table records individual physical activities, including start and end times, activity name, distance, and duration.Its schema also includes elevation gain, average heart rate, calories, steps, and active-zone minutes.Source: §C.2.2 · p. 63

C.2.3. Population Data Table

  • The population data table organizes age- and gender-specific percentiles and includes measures such as resting heart rate, heart-rate variability, steps, and sleep.Its listed measures also include active-zone minutes, sleep stages, and stress-management scores.Source: §C.2.3 · p. 64

C.3.1. Base Gemini Prompt Template

  • The base Gemini prompt asks an expert data scientist to generate a step-by-step analysis approach with one discussion and one approach field.Source: §C.3.1 · p. 66

C.3.2. Approach Generation Prompt Template

  • The approach-generation prompt requires a feasibility and ambiguity discussion, operationalized terms, and a precise plan based only on the query and available schemas.For answerable queries, steps specify timeframe, data selection, transformations, sufficiency and missing-data checks, statistics and assumptions, and output; unanswerable queries require a helpful explanation. Plans must avoid assumptions, code, and interpretation.Source: §C.3.2 · pp. 66–67

C.3.3. Base Gemini Code Generation Prompt Template

  • The base code-generation prompt supplies dataframe schemas and a docstring, then directs the model to generate Python code without reproducing the docstring or comments.Source: §C.3.3 · p. 68

C.3.4. DS Agent Code Generation Prompt Template

  • The code-generation prompt supplies DataFrame schemas, a data summary, and an analysis approach, and requires idiomatic pandas code that uses the datetime index and returns only executable code.The instructions require necessary imports, prohibit treating datetime as a column unless the schema says otherwise, and forbid comments or the docstring in the output.Source: §C.3.4 · pp. 68–69

C.3.5. Code Debug Prompt Template

  • The debugging prompt asks the model to correct generated code using the original goal and execution results while preserving intent and structure and avoiding unrelated changes.It requests only the complete corrected Python code block and can incorporate optional code-update feedback.Source: §C.3.5 · p. 70

C.4.1. Task 1 Rubrics: Analysis Plan Generation

  • The analysis-plan rubric evaluates timeframe, transformations, data sufficiency, summary statistics, statistical tests, and alignment with user intent.It checks whether timeframe and transformations are clear and reasonable, data checks cover relevant rows and column cells, and statistical methods are reproducible and appropriate to distributional assumptions.Source: §C.4.1 · p. 71
  • After removing 7 low-agreement or rarely applicable items from 33, the final rubric used 26 items, with human-human agreement of 80.2% and Bennett’s S of 0.622.The original 33-item rubric had 78.4% agreement and Bennett’s S of 0.574 across 141 query-plan pairs (6,157 items); the score could deduct up to 16 points, with performance calculated as the non-deducted share.Source: §C.4.1 · p. 71
  • The rubric cautions evaluators not to make extra assumptions.Source: §C.4.1 · p. 73
  • The rubric checks whether a statistical test would improve the analysis when needed and whether the data distribution meets the assumptions of a test that is used.Source: §C.4.1 · p. 73
  • Overall, evaluators assess whether the analysis approach aligns with the user’s intent and provides a satisfactory answer.Source: §C.4.1 · p. 73

C.4.2. Task 2 Unit Tests: Code Generation and Execution

  • The 25 Task 2 test suites include 173 unit tests assessing limited-data handling, transformations, statistical methods, and whether code ultimately answers the user’s question.Examples check for an “Insufficient data” response when a comparison group has fewer than 30 days and verify test selection for normally and non-normally distributed data.Source: §C.4.2 · pp. 73–74

C.5. DS Agent Task 1 Auto-Rater Development and Reliability … D.2. Prompt Setups

The supplementary material reports the DS auto-rater’s agreement and reliability, describes evaluation-task setups, and documents the DE agent’s preamble and task prompts. The DE prompts emphasize personalized, contextualized responses and specify formats for multiple-choice, differential-diagnosis, contextualization, and multimodal-summary tasks.

C.5. DS Agent Task 1 Auto-Rater Development and Reliability

  • The autorater achieved 82.4% agreement with experts, slightly exceeding human-human agreement, 86.9% agreement with the human majority, and ICC(3,1)=0.838 for analysis-plan scoring.The ICC was calculated from human-assigned point deductions and deductions inferred from autorater outputs; the authors describe the resulting evaluations as moderately reliable and suitable for scaling.Source: §C.5 · p. 75, Table S4 · p. 75

C.6.1. Task 1: Analysis Plan Generation; C.6.2. Task 2: Code Generation and Execution

  • Tables S6 and S7 describe analysis-plan scores across 99 data-science queries and code-generation error rates, respectively, with statistical tests and uncertainty procedures specified.The S6 caption reports means and standard errors with Wilcoxon signed-rank tests and rank-biserial effect sizes; S7 describes bootstrap standard errors and tests for error comparisons.Source: Table S6 · p. 77, Table S7 · p. 77

D.1. Conversation Examples

  • Figure S4 provides an example multi-turn conversation with the DE agent.The supplied passage identifies the figure’s scope but does not provide conversation details.Source: Figure S4 · p. 78

D.2.1. DE Agent Preamble; D.2.2. Base Model Preamble

  • The DE preamble calls for authoritative, non-hallucinated and personalized health analysis, feasible actionable recommendations, and coordination with the data-science or coaching agent when appropriate.It also directs the agent to use tools or request missing data only when necessary and provide source links when using web search; the base-model preamble specifies health expertise and stepwise reasoning.Source: §D.2.1 · pp. 79–81, §D.2.2 · p. 83

D.2.3. Prompt for Task 1 (MCQ) for All Models/Agents

  • For Task 1, all models and agents are prompted to give the correct multiple-choice letter first, followed by a brief explanation when needed; five-option questions add E..Source: §D.2.3 · p. 83

D.2.4. Prompt for Task 2 (DDx) for All Models/Agents

  • For differential diagnosis, models must explain their approach and return ten ranked diagnoses; the autoevaluator accepts synonyms and outputs the correct diagnosis’s rank or NaN for hit@k scoring.The evaluator is instructed to identify the underlying condition and its position in the ranked response.Source: §D.2.4 · p. 84

D.2.5. Prompt for Task 3 (Contextualization and Personalization) to All Agents/Model

  • Task 3 prompts all agents and models for comprehensive, contextualized answers with authoritative citations when appropriate.Source: §D.2.5 · p. 85

D.2.6. Prompt for Task 4 (Summarization of Multimodal Health Data) to All Agents/Model

  • Task 4 supplies baseline survey information, wearable-derived biomarker averages, standard deviations and medians from the 90 days before blood collection, and blood biomarker values for health-data summaries.Source: §D.2.6 · p. 85

D.3. DE Agent Evaluation Rubrics Details · D.4. Additional DE Agent Evaluation Results

The evaluation rubrics assess generalists’ judgments of contextualized health-query responses and clinicians’ judgments of multimodal health summaries, emphasizing personalization, clarity, factual accuracy, clinical relevance, cross-modal reasoning, and trustworthy references.

D.3. DE Agent Evaluation Rubrics Details

  • For Task 3, generalists compare whether responses use the query’s personal context to provide fully relevant, tailored answers.The rubric considers details such as age and existing conditions.Source: §D.3 · p. 87
  • Generalists also compare whether responses define technical or medical jargon well enough to be easier for non-experts to understand.The rubric asks which response uses fewer undefined terms.Source: §D.3 · p. 87
  • Clinician rubrics check whether summaries correctly report test values and ranges and match diagnoses to the source data.They flag misassigned values, incorrect ranges, and ignored patient conditions.Source: §D.3 · p. 89
  • Clinicians assess whether summaries highlight clinically significant findings, risks, and areas of concern.The rubric rewards identifying concerns in the user’s data and penalizes omissions of critical information.Source: §D.3 · p. 89
  • Clinicians assess whether summaries connect and reason across data modalities rather than mentioning multiple sources without integrating them.The example links glycemic results with a sedentary lifestyle to contextualize a recommendation for activity.Source: §D.3 · p. 89
  • Clinicians assess whether medical summaries are comprehensive, clear to a college-level reader, and explain terminology rather than leaving abbreviations undefined.A summary need not include every datum, but should explain what it presents.Source: §D.3 · p. 89
  • Clinicians assess whether references are relevant, accessible, and verifiable, including whether links resolve to pages matching the cited context.Broken or unrelated links count against a response.Source: §D.3 · p. 90
  • Clinicians judge which response is more comprehensive or personalized and which they prefer for trustworthiness.The rubric emphasizes a satisfying account that acknowledges the user’s data.Source: §D.3 · p. 90

D.4.1. Task 1: Answering Medically-Relevant Multiple Choice Questions

The DE agent outperformed base Gemini across four domains of expert-level health questions, with the largest reported improvement on Fitness MCQs.

D.4.1. Task 1: Answering Medically-Relevant Multiple Choice Questions

  • Sleep accuracy was 88.0% (±1.3%) for the DE agent versus 86.4% (±1.4%) for Gemini; Endocrinology accuracy was 82.7% (±1.9%) versus 80.7% (±2.0%).The reported absolute improvements were 1.58% in Sleep and 2.01% in Endocrinology.Source: §D.4.1 · p. 91
  • In Cardiology, DE agent accuracy was 73.7% (±1.8%), compared with 73.2% (±1.9%) for base Gemini.Source: §D.4.1 · p. 91
  • Fitness accuracy was 89.7% (±1.3%) for the DE agent versus 86.2% (±1.5%) for base Gemini, the largest improvement (p< 0.001).Source: §D.4.1 · p. 91, Table S10 · p. 91
  • Across more than 2,000 questions in Cardiology, Endocrinology, Sleep, and Activity, the DE agent achieved higher accuracy than base Gemini in every domain.The evaluation used four curated MCQ datasets.Source: Table S10 · p. 91, §D.4.1 · p. 91, Figure S5 · p. 92

D.4.2. Task 2: Diagnostic Conversations … E.4.1. Task 1: Health Coach Agent Capability from End-Users’ Perspectives

This section reports the DE Agent’s differential-diagnosis results and specifies the HC Agent’s coaching behavior and evaluation criteria. It also presents the health coach agent’s rubric-based performance scores, side-by-side win rates, and statistical tests for each measure.

D.4.2. Task 2: Diagnostic Conversations

  • The diagnostic benchmark drew on 2,000 recruited users across 39 self-reported condition categories, grounding differential diagnosis in patient-language presentations.The listed conditions span common symptoms and concerns, including pain, respiratory problems, and anxiety.Source: §D.4.2 · pp. 93–94
  • DE Agent top-1, top-5, and top-10 accuracy was 46.1%, 75.6%, and 84.5%, respectively, exceeding the DDx Agent’s 41.4%, 71.4%, and 81.1%.The comparison uses top-k inclusion of the ground-truth diagnosis; Figure S6 shows the accuracy comparison.Source: §D.4.2 · p. 94, Figure S6 · p. 95

E.2.2. HC Agent Role Definition for Personalized Module

  • The HC role prompt directs the coach to uncover the user’s goal, motivation, and constraints, ask about preferences, and guide the user toward a recommendation without acting as an authority.It also instructs the coach to use a casual, motivational tone, avoid assumptions, and periodically summarize the plan.Source: §E.2.2 · pp. 101–102

E.2.3. HC Agent Recommendation Module

  • The recommendation module gates advice on whether the conversation has established why the user wants the goal, what it means to them, their constraints, and preferred actions.A separate two-call conclusion module determines whether the conversation has ended and, if so, generates a closing summary and encouragement.Source: §E.2.3 · p. 103, §E.2.4 · p. 103

E.3.1. User Case Selection

  • Before conversations, end-users selected an everyday-life scenario and were instructed not to share personal health records.The selection prompt asked participants to choose a relatable scenario before interacting with five agents.Source: Table S17 · p. 104

E.3.3. Task 2: Expert Evaluation Rubrics

  • The expert evaluation rubrics cover goal reframing and SMART goal-setting alongside coaching qualities such as personalization and a clear wrap-up.The rubrics also assess friendliness, support, clarity, rapport, information gathering, empowerment, and capturing the user’s concern.Source: §E.3.3 · pp. 110, 112

E.4.1. Task 1: Health Coach Agent Capability from End-Users’ Perspectives

  • End-user rubric performance averaged scores across CEMI, CUQ, SMART, and WAI for 31 participants; GLMM with a Gamma link tested differences.Because each end-user interacted with the agents individually, inter-rater reliability was not applicable.Source: Table S24 · p. 115
  • In side-by-side comparisons by 31 end-users, performance was measured by win rate, with standard errors bootstrapped over 1,000 resamples and binomial GLMM tests.Inter-rater reliability was not applicable.Source: Table S25 · p. 115

E.4.2. Task 2: Validation of Health Coach Agent Design with Expert Evaluation … F.4. PHA Evaluation Results

This section describes the evaluation of the Health Coach Agent and personal health agent conversations, including PHA orchestration prompts, single- and multi-agent baselines, and end-user and expert procedures. It also reports expert evaluation measures and defines behavioral rubrics for assessing the Health Coach Agent’s interpretation, communication, personalization, integration, and safety-related judgment.

E.4.2. Task 2: Validation of Health Coach Agent Design with Expert Evaluation

  • Experts (N=3) rated HC rubrics with good inter-rater reliability for both base Gemini (ICC(3,1)=0.794) and the HC agent (ICC(3,1)=0.690); side-by-side win rates also had good IRR (Fleiss’ κ=0.640).Rubric performance averaged six themes and structured measures; side-by-side standard errors used 1,000 bootstrap samples.Source: Table S26 · p. 116, Table S27 · p. 117

F.2. Prompt Setups

  • PHA prompts include the complete conversation history under a [CONVERSATION] tag, and the section summarizes prompts for PHA and two baselines: single-agent PHIA and a parallel multi-agent system.Source: §F.2 · p. 118

F.2.1. PHA Orchestrator Role Definition and Integration of Other Agents

  • The orchestrator organizes the conversation and selects among data science, domain expert, and health coach agents according to whether a query needs personal-data analysis, domain knowledge, or coaching.Source: §F.2.1 · p. 118

F.2.2. PHA Orchestrator Task Assignment

  • For task assignment, the orchestrator matches queries to a main agent, supporting agents, and workflow, adds data science support when personal data may be relevant, and falls back to basic Gemini if no example matches.It first checks non-health corner cases, then applies the workflow examples.Source: §F.2.2 · p. 123

F.2.3. PHA Orchestrator Question Rephrase

  • After assigning agents, PHA decomposes and rephrases the user’s question into clear, agent-specific questions that support the main agent’s response.Source: §F.2.3 · p. 126

F.2.4. PHA Reflection Prompt

  • PHA reflects on the main agent’s response and can ask supporting agents targeted follow-up questions when their answers could improve it, avoiding questions that personal data can answer.Source: §F.2.4 · p. 127

F.2.5. Singe Agent Baseline PHIA Prompts

  • The PHIA single-agent baseline uses a ReAct prompt with Python and search tools, user data when relevant, and a conversational multi-turn question-and-state format.Source: §F.2.5 · pp. 128, 130
  • The single-agent baseline prompt instructs the agent to include its reasoning appropriately in the final-answer function call.The prompt names this function finish().Source: §F.2.5 · p. 130

F.2.6. Parallel Multi-Agent Baseline: Prompts

  • The parallel baseline runs the DS, DE, and HC agents independently on the same query, then has an orchestrator synthesize their outputs with the conversation history.Unlike PHA, this baseline uses no task assignment or agent collaboration.Source: §F.2.6 · pp. 130–131

F.3.1. Task 1: End-User Evaluation Rubrics

  • End-user generalists adopt each user persona, read conversations with three agents, and rate the interaction from that user's perspective, focusing on conversation flow rather than numeric accuracy or formatting.The persona stays generally consistent across conversations, while persona data may be historical and differ from data in the scripts.Source: §F.3.1 · p. 131
  • End-user rubrics cover data relevance, useful health knowledge, contextualized interpretation, supportive tone, preference-aware flow, and personalized behavior-change guidance.They also assess confidence-building, alignment with personal priorities and barriers, and whether suggestions are individualized to personal data and grounded in health knowledge.Source: §F.3.1 · pp. 133–135

F.3.2. Task 2: Expert Evaluation Rubrics

  • Health experts review the user persona and each agent's turns, then assess that specific interaction professionally; they compare conversations under a generally consistent persona and disregard minor numeric differences and formatting.Persona data is historical context and may not exactly match data in the conversations.Source: §F.3.2 · p. 138
  • Expert rubrics assess overall response quality and user-specific effectiveness, appropriate use of available data, and health information accuracy, safety, relevance, and interpretation.Available data includes basic characteristics, Fitbit step, sleep, and activity records, and blood test results.Source: §F.3.2 · pp. 139–141
  • The rubric assesses whether health-data interpretation is accurate, contextualized, appropriate, and useful to the user.Interpretation is rated from fundamentally flawed to correct and contextualized; usefulness ranges from potentially misleading to practical value.Source: §F.3.2 · p. 141
  • Coaching criteria cover welcoming communication, empathy, user empowerment, clarity, elicitation of preferences and barriers, and suggestion timing.They also assess responsiveness to feedback and questions, and guidance toward contextually appropriate SMART goals.Source: §F.3.2 · pp. 141–143
  • Personalization rubrics assess whether coaching draws appropriately on the user’s health data and integrates evidence-based health recommendations into behavior-change dialogue.Goal-setting is also evaluated for alignment with health principles, SMART criteria, and the user’s context.Source: §F.3.2 · pp. 143–144
  • Additional criteria evaluate response coherence and synthesis across user data, health knowledge, and coaching strategies.The rubrics also check whether the agent recognizes and responds appropriately to red flags and asks necessary clarifying questions.Source: §F.3.2 · pp. 144–145
  • The presented rubric table aggregates one or more raw questions for each rubric.The interpretation rubric distinguishes five levels, from incorrect to correct and contextualized.Source: §F.3.2 · p. 141, Table S31 · p. 146

F.4.1. Task 1: Multi-agent Conversation Experience for End-Users · F.4.2. Task 2: Multi-agent Conversation Experience for Experts

The two conversation-experience evaluations assessed end-user and expert ratings using rubric scores and side-by-side rankings. The passages report inter-rater reliability and agreement, but do not provide comparative performance results.

F.4.1. Task 1: Multi-agent Conversation Experience for End-Users

  • End-users’ rubric ratings had excellent inter-rater reliability: single-agent PHIA ICC(3,1)=0.890, parallel multi-agent ICC(3,1)=0.812, and PHA ICC(3,1)=0.825.Performance averaged rubric scores across the DS, DE, and HC agents and their combinations; statistical tests used a Gamma-link GLMM and post-hoc Tukey’s HSD.Source: Table S32 · p. 147
  • The end-user rubric evaluation averaged scores across DS, DE, and HC agents and their combinations.It used a Gamma-link GLMM for statistical tests and post-hoc Tukey’s HSD for pairwise comparisons.Source: Table S32 · p. 147
  • End-users’ side-by-side evaluation measured ranking rates and reported moderate inter-rater reliability (Fleiss’ κ=0.468).Standard errors used 1,000 bootstrap samples; Friedman tests were followed by Wilcoxon rank-sum tests with Holm-Bonferroni correction.Source: Table S33 · p. 147

F.4.2. Task 2: Multi-agent Conversation Experience for Experts

  • Experts’ rubric ratings had moderate-to-good inter-rater reliability: single-agent PHIA ICC(3,1)=0.630, parallel multi-agent ICC(3,1)=0.545, and PHA ICC(3,1)=0.690.Performance averaged rubric scores across the DS, DE, and HC agents and their combinations; statistical tests used a Gamma-link GLMM and post-hoc Tukey’s HSD.Source: Table S34 · p. 148
  • Experts’ side-by-side evaluation reported an 81.5% agreement rate.Performance was measured by ranking rate; standard errors used 1,000 bootstrap samples, and Friedman tests were followed by Wilcoxon rank-sum tests with Holm-Bonferroni correction.Source: Table S35 · p. 148
  • The expert side-by-side evaluation compared ranking rates using Friedman tests and post-hoc Wilcoxon rank-sum tests with Holm-Bonferroni correction.Standard errors were calculated with 1,000 bootstrap samples with replacement.Source: Table S35 · p. 148

Related papers

Loading 2508.20148v2…