PaperlayerView exact revision on arXiv

Source-linked AI summary

SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment

Joseph Breda, Fadi Yousif, Beszel Hawkins, Marinela Cotoi, Miao Liu, Ray Luo, Po-Hsuan Cameron Chen, Mike Schaekermann, Samuel Schmidgall, Xin Liu, Girish Narayanswamy, Samuel Solomon, Maxwell A. Xu, Xiaoran Fan, Longfei Shangguan, Anran Wang, Bhavna Daryani, Buddy Herkenham, Cara Tan, Mark Malhotra, Shwetak Patel, John B. Hernandez, Quang Duong, Yun Liu, Zach Wasson, Dimitrios Antos, Bob Lou, Matthew Thompson, Jonathan Richina, Anupam Pathak, Nichole Young-Lin, Jake Sunshine, Daniel McDuff

arXiv:2605.04012v2cs.AI

TL;DR

Prior symptom-assessment studies did not represent everyday patient conversations or population distributions of symptoms and illnesses. In a national-scale naturalistic deployment, SymptomAI DDx lists outperformed clinician DDx lists in all conditions on blinded preference and Top-5 accuracy against patient-relayed diagnoses.

  • Problem

    Prior symptom-assessment studies did not represent everyday patient conversations or population distributions of symptoms and illnesses.

    Source: §1 · p. 3

  • Method

    The team deployed SymptomAI in a broad national sample for naturalistic symptom interviews, collected participant-reported diagnoses, and had clinicians evaluate a subset.

    Source: §1 · p. 4, §3 · p. 10

  • Results

    SymptomAI DDx lists outperformed clinician DDx lists in all conditions on blinded clinician preference and Top-5 accuracy against patient-relayed diagnoses.

    Source: §3 · p. 8

  • Takeaways & Limitations

    The findings indicate an opportunity for more accurate symptom-based LLM diagnoses through organized interviews rather than entirely user-guided conversations.

    Source: §5 · p. 13

    Clinician DDx were based on AI-led conversations rather than clinician-directed interviews, limiting a fully controlled end-to-end comparison.

    Source: §4 · p. 12

Abstract

from arXiv · show

Language models excel at diagnostic assessments on curated medical case-studies and vignettes, performing on par with, or better than, clinical professionals. However, existing studies focus on complex scenarios with rich context making it difficult to draw conclusions about how these systems perform for patients reporting symptoms in everyday life. We deployed SymptomAI, a set of conversational AI agents for end-to-end patient interviewing and differential diagnosis (DDx), via the Fitbit app in a study that randomized participants (N=13,917) to interact with five AI agents. This corpus captures diverse communication and a realistic distribution of illnesses from a real world population. A subset of 1,228 participants reported a clinician-provided diagnosis, and 517 of these were further evaluated by a panel of clinicians during over 250 hours of annotation. SymptomAI DDx were significantly more accurate (OR = 2.56, p < 0.001) than those from independent clinicians given the same dialogue in a blinded randomized comparison. Moreover, agentic strategies which conduct a dedicated symptom interview that elicit additional symptom information before providing a diagnosis, perform substantially better than baseline, user-guided conversations (p < 0.001). An auxiliary analysis on 1,509 conversations from a general US population panel validated that these results generalize beyond wearable device users. We used SymptomAI diagnoses as labels for all 13,917 participants to analyze over 500,000 days of wearable metrics across nearly 400 unique conditions. We identified strong associations between acute infections and physiological shifts (e.g., OR > 7 for influenza). While limited by self-reported ground truth, these results demonstrate the benefits of a dedicated and complete symptom interview compared to a user-guided symptom discussion, which is the default of most consumer LLMs.

1. Introduction

The introduction motivates evaluating conversational AI symptom assessment with real users and health needs, then presents SymptomAI as a way to study diagnostic performance and wearable-signal associations at scale.

1. Introduction

  • Traditional online symptom checkers have diagnostic accuracies of 20–40%, despite serving as a primary entry point to downstream medical care.The introduction frames the broader gap as a lack of large-scale evaluation of conversational AI for layperson symptom assessment using clinical measures such as diagnostic accuracy.Source: §1 · pp. 2–3
  • 94.5% to 34.5%: layperson involvement in communicating context reduced diagnostic accuracy relative to applying AI directly to clinical vignettes, underscoring the need for naturalistic evaluation.The passage attributes the decline largely to incomplete or misrepresented information provided by nonexperts and notes that large-scale diagnostic-accuracy evaluation with laypeople had not been demonstrated.Source: §1 · p. 3
  • 13,917 participants generated multi-turn symptom conversations after randomization across five prompting strategies, from structured history-taking to a dynamic agent.Participants reported healthcare-provider diagnoses; three board-certified family physicians independently assessed top-five DDx accuracy for clinicians and SymptomAI while blinded to authorship and participant diagnoses.Source: Figure 1 · p. 2, §1 · pp. 3–4
  • SymptomAI diagnoses, particularly for acute respiratory infections, shared trends with wearable biosignals in a cohort-wide analysis of nearly 400 conditions and over 500,000 days of data.The authors used SymptomAI’s top diagnosis as a reference label across all 13,917 participants, enabling a phenome-wide association study at a scale they say would otherwise have been infeasible.Source: Figure 1 · p. 2, §1 · p. 4

2. Results

SymptomAI’s top-5 differential-diagnosis accuracy exceeded that of baseline clinicians, and information-eliciting interviews outperformed user-guided prompting. Additional analyses found similar accuracy in a general-population panel and wearable-signal shifts around respiratory-infection symptom reporting.

2.1. Conversations That Elicit More Information Outperform User-Guided Conversations

  • 27.57% higher average accuracy resulted from information-eliciting interview strategies than from base user-guided prompting.Each eliciting arm independently outperformed the base condition (arms 2–5: p = 0.003, p < 0.001, p = 0.003, and p = 0.022, respectively); dynamic strategies and canonical-question strategies had comparable accuracy (72.07% vs. 77.51%, p = 0.235).Source: §2.1 · pp. 4–5

2.3. SymptomAI DDx is More Accurate than Clinicians

  • Median OR 2.56 (95% CI Cohen’s g [0.18, 0.26], p < 0.001) for SymptomAI’s higher top-5 DDx accuracy over baseline clinicians.The advantage also held for conversations rated highest in quality, where clinicians had complete clinical information for comparison.Source: §2.3 · p. 5, Figure 2 · p. 6

2.2. Clinical Experts Prefer SymptomAI Over Clinician-Generated Differential Diagnoses

  • Over 50% of cases ranked SymptomAI’s DDx as the best option, favoring it over chance (OR = 2.34, n = 517, p < 0.001).Blinded clinical raters ranked SymptomAI and two independent clinicians’ DDx; preference for SymptomAI was most pronounced in conversations rated highest quality.Source: §2.2 · p. 5, Figure 2 · p. 6

2.4. Robustness of SymptomAI on Low Information Conversations

  • For conversations where clinicians felt neutral or not confident in their own DDx, SymptomAI significantly outperformed clinicians (McNemar’s test, p < 0.001).The systems performed similarly on conversations where clinicians felt confident in their own DDx.Source: Figure 2 · p. 6, §2.4 · p. 7

2.5. Representativeness of the Study Population

  • 72.6% top-5 accuracy in the general-population panel compared with 77.7% in the SymptomAI study population supports similar performance across these study populations.The panel included 1,509 people and had a significantly different illness distribution; the clinical-evaluation subsample showed no practically meaningful demographic shifts, only a small diagnosis-category shift, and diagnosis self-report prediction had AUC = 0.591.Source: §2.5 · p. 7

2.6. Diagnoses from SymptomAI Correlate with Physiological Biosignal Onset

  • 1,546 participants diagnosed by SymptomAI with respiratory infection showed distinct wearable-biosignal shifts in the days leading up to symptom reporting.Across eight biosignals, significant shifts were most evident for respiratory and circulatory illnesses, with respiratory shifts concentrated in recent days before engagement; the respiratory cohort excluded non-infectious conditions such as allergic rhinitis and COPD.Source: §2.6 · pp. 7–8

3. Discussion

In a population-scale, real-world deployment, SymptomAI’s diagnostic lists outperformed clinician lists, and dedicated symptom interviews performed better than user-guided conversations. Wearable signals also shifted around symptom-checker use, suggesting potential physiological context for diagnoses and earlier illness detection.

Discussion

  • Symptom assessment accuracy degrades when agents do not ask questions to elicit more information, a limitation of the user-guided approach used by major consumer-facing LLMs.Prior vignette-based results also fell when laypeople relayed the information conversationally to an AI, underscoring the need for in-the-wild evaluation.Source: §3 · p. 8
  • SymptomAI DDx lists outperformed clinicians’ across all conditions in both blind preference and Top-5 accuracy, with a larger advantage when clinicians were less confident.The comparisons used ranked preference while clinicians were blind to ground truth and Top-5 accuracy when given patient-relayed diagnoses.Source: §3 · p. 8
  • Strong performance across varied conditions and communication styles, including atypical or confounded cases, suggests symptom reasoning beyond matching common presentations to likely diagnoses.The authors note that broad distributional priors may explain textbook presentations but may not alone explain performance on atypical or confounded cases.Source: §3 · p. 9
  • The study evaluated symptom assessment at population scale with naturally occurring user-reported symptoms and illnesses, rather than relying only on standardized clinical vignettes.The authors describe it as the largest in-the-wild evaluation of generative AI symptom interviews and diagnostic assessment to date.Source: §3 · pp. 8–10
  • Wearable measures of cardiovascular function, respiration, sleep, skin temperature, and activity shifted in the days before conversations, potentially supplementing diagnosis with physiological context.For respiratory infections, time awake during sleep peaked around the conversation date, while non-REM nighttime heart rate and heart-rate variability changed steeply and could be passive early-warning signals.Source: §3 · pp. 10–11, Figure 5 · p. 11
  • Immediate access to symptom interviews may capture onset while symptoms are fresh and could support earlier treatment or behavior changes that reduce infectious-disease transmission.The authors suggest combining vital-sign changes with symptom interviewing and diagnosis to prompt check-ins before people would otherwise seek medical guidance.Source: §3 · p. 12

4. Limitations

The evaluation is qualified by uncertainty in diagnostic labels, a clinician comparison based on AI-led conversations, and symptom reports that vary in timing and may require clinical confirmation. These limitations constrain how the performance results should be interpreted.

4. Limitations

  • 10–15% of clinical encounters result in diagnostic error, underscoring the inherent ambiguity of symptom assessment.The authors note that their analysis is likewise affected by this ambiguity.Source: §4 · p. 12
  • Second- or third-hand diagnosis reports may be inaccurate, outdated, or misrepresented, and scaling evaluation to tens of thousands of people introduces noise despite filtering.Evolving conditions may also mean a diagnosis has changed since it was reported.Source: §4 · p. 12
  • The clinician comparator assessed differential diagnoses from participant–SymptomAI conversations, not interviews conducted by clinicians, limiting a controlled end-to-end comparison.SymptomAI still outperformed clinicians on high-quality interactions with requisite diagnostic information (Figure 2e); prior work found conversational AI elicited symptom information with detail and accuracy comparable to clinicians.Source: §4 · p. 12
  • Symptom reports are time-specific snapshots, and uncontrolled reporting timing, the need for confirmatory tests, and possible chronic-illness misdiagnosis limit assessment.The authors suggest future targeted studies at specific stages of illness and longitudinal assessment of specific populations.Source: §4 · p. 12

5. Conclusion

SymptomAI’s organized symptom interviews produced more accurate differential diagnoses than user-guided conversations and enabled population-scale analyses of wearable signals associated with reported illness. The study also releases de-identified research data under controlled access, provides analysis code, and emphasizes that the system is investigational rather than clinically validated.

5. Conclusion

  • SymptomAI’s differential diagnoses were more accurate than those from board-certified clinicians, and organized interviews outperformed user-guided symptom conversations.The study also used SymptomAI diagnoses to analyze population-scale associations between wearable physiological signals and reported illness.Source: §5 · p. 13

Data availability

  • De-identified conversation and clinician-evaluation data are available to qualified researchers under controlled access, while raw wearable data are not released.Some data streams could not be fully anonymized, so the researchers developed policies and infrastructure to protect privacy and govern data use.Source: Data availability · p. 13

Code Availability

  • The authors provide code implementations for data analysis, plotting, and supplementary analysis.Source: Code Availability · p. 13

Disclaimer

  • SymptomAI is an investigational research prototype, not a medical device or validated diagnostic tool, and its model-generated labels do not establish clinical diagnoses.It is not commercially available and does not imply a commitment to a future product roadmap.Source: Disclaimer · p. 13

A. Methods … A.3. Agent Arm Designs

SymptomAI was evaluated in a randomized Fitbit-app study of five language-model interview designs, combining symptom conversations with candidate diagnoses and assessments of diagnostic performance. The methods describe the in-app study procedure and the distinct interview strategies assigned to participants.

A.1. SymptomAI Study Objective

  • The study randomized 13,917 participants to five agent arms and assessed SymptomAI differential diagnoses against clinician-generated diagnoses and clinician preferences.The study hypothesized that agents prompted to ask follow-up questions would improve top-5 accuracy over unprompted, user-guided symptom conversations.Source: §A.1 · p. 14

A.2. Randomized In-Situ Population Deployment

  • Consent characterized the study as a non-interventional observational survey for research and model benchmarking, not medical care.Participants consented electronically before study procedures and were told the research prototype did not establish a provider-patient relationship.Source: §A.2 · pp. 14–15
  • After consent, participants described symptoms and interacted with their assigned agent, saw candidate diagnoses, and could complete a satisfaction survey and report a clinician diagnosis.Participants without a provider diagnosis could receive a follow-up notification two weeks later; there was no limit on repeat use while enrolled.Source: Figure 6 · p. 14, §A.2 · p. 15
  • Of 40,000 enrolled Fitbit users, 13,917 completed at least one conversation, and 1,228 were linked to a patient-reported diagnosis.Recruitment was conducted through the Fitbit app among US users; the study ran from June 2025 until April 17, 2026.Source: §A.2 · pp. 14–15

A.3. Agent Arm Designs

  • Arm 1 approximated ordinary user-guided chat with only restrictions against non-health topics and an instruction to provide a differential diagnosis list.The five arms varied the level of instruction for conducting the symptom interview.Source: §A.3 · p. 15
  • Arm 2 asked fixed canonical history questions in up to six turns, while Arm 3 sought the same information with flexible phrasing and no prescribed questions.The queried information included symptom location, onset, severity, quality, frequency, modifying factors, and preexisting risk factors.Source: §A.3 · p. 15
  • Arm 4 dynamically asked narrowing questions and supplied a differential diagnosis at every turn; Arm 5 used a similar dynamic interview but omitted diagnoses during the turns.The Arm 4 prompt was designed with a prompt optimizer (Sayres et al., 2026).Source: §A.3 · pp. 15–16

A.4. Evaluation

The evaluation built a clinician-annotated sample of diagnosed conversations to compare SymptomAI DDx with participant-reported diagnoses. It also defined accuracy measures for the annotated sample and the broader diagnosed cohort, and used SymptomAI’s top diagnosis as a silver-standard label for wearable-trend analyses.

A.4. Evaluation

  • Three clinicians annotated diagnosed conversations to establish a baseline clinical DDx and ground participant symptom reports in expert feedback.The study prioritized complete conversations ending in a SymptomAI DDx.Source: §A.4 · p. 16
  • The clinical evaluation began with 517 eligible conversation–diagnosis pairs and reached 1,228 by study end after filtering for workable conversations and appropriate reported diagnoses.Workable conversations had at least 10 cumulative user-message words and passed a Gemini quality screen; responses to the HCP-diagnosis survey were also screened for completeness and appropriateness.Source: §A.4 · p. 16
  • For the clinical evaluation sample, accuracy was the percentage of cases where a clinician identified the participant-reported diagnosis as clinically identical to a candidate in the top-five DDx.The metric therefore assessed whether the reported diagnosis appeared among the top-five candidates, as judged by a clinical rater.Source: §A.4 · p. 16
  • Across cases with a participant-reported diagnosis, accuracy was the percentage where an LLM verifier found that diagnosis among SymptomAI’s top five, with auto-rater validation against clinical raters in the evaluation subset.This measure included the clinical evaluation sample as well as other cases with a participant-reported diagnosis.Source: §A.4 · p. 16
  • For wearable-biosignal analyses, the top-1 SymptomAI DDx was treated as a “silver standard” label after validation in the clinical evaluation.All cases, including those with a patient-reported diagnosis, were used for the wearable biosignal association analysis.Source: §A.4 · pp. 16, 18

A.4.1. Clinical Evaluation Tasks … Q9. On a scale of 1–5, how similar are these DDx lists to each other?

The clinical evaluation used two sequential tasks: clinicians first generated blinded baseline differentials, then a blinded rater compared those lists with SymptomAI’s and assessed related questions. The merged sections also specify clinician survey prompts and evaluation questions about chatbot behavior, rankings, and differential-list similarity.

A.4.1. Clinical Evaluation Tasks

  • In Task 1, clinicians independently produced best-effort differentials from conversations with SymptomAI’s diagnosis redacted, assessed their confidence and conversation quality, and reviewed conversations in round-robin pairings.Each clinician covered two-thirds of the conversation pool, with each third reviewed by a different clinician pairing.Source: §A.4.1 · pp. 19–21, Figure 10 · p. 20
  • In Task 2, a blinded clinical rater ranked three standardized, randomly ordered differentials—two clinician lists and SymptomAI’s—then assessed SymptomAI’s completeness and appropriateness after its list was revealed.The rater also reviewed the full conversation for the latter assessment.Source: Figure 10 · p. 20
  • Clinicians assessed whether self-reported diagnoses matched candidate diagnoses in each differential and whether chatbot responses could cause harm.They also judged diagnosis appropriateness, support from reported symptoms, and whether medical information was appropriate given the diagnosis.Source: §A.4.1 · p. 21

A.4.2. Survey Questions

  • Task 1 survey prompts asked clinicians to provide five candidate diagnoses, rate confidence, judge whether the conversation supplied enough information, and suggest crucial follow-up questions.The survey also asked how much additional information clinicians would need to become very confident.Source: §A.4.2 · p. 23

Q6. Any other comments about this conversation or the chatbot’s behavior?

  • Task 2 questions were delivered in ordered batches with relevant transcripts and differential lists, while the redacted conversation remained shared context across questions.This sequencing was intended to obtain some answers before raters knew the patient’s ground-truth outcome.Source: Q6. Any other comments about this… · p. 24

Q8. Please provide any comments about the ranking.

  • The ranking comments prompt asked raters to note when two differential lists were functionally the same and the third differed.The provided instruction gives this as the circumstance for using the comment field.Source: Q8. Please provide any comments about… · p. 24

Q9. On a scale of 1–5, how similar are these DDx lists to each other?

  • The SymptomAI differential evaluation asked raters to rate list similarity, with response choices ranging from all reasonable candidates present to major candidates missing.The next task section revealed the self-reported diagnosis for its assessment.Source: Q9. On a scale of 1–5, how similar are… · pp. 25–26

Q14. Please provide details. … Q41. I know how to find helpful health resources on the Internet.

The appendix documents an auxiliary general-population study and the survey and evaluation instruments used to collect symptom histories, diagnoses, and health-information behaviors. Its clinical review prompts assess diagnosis support, DDx placement, and potential harm.

Q14. Please provide details.

  • Reviewers rated confidence on a 1–7 scale that information supported the self-reported diagnosis, judged whether it was misclassified or inconsistent with symptoms, and identified missing information when support was insufficient.They then checked all three DDx lists for the diagnosis, marking its position (1–5) or “Not Present”; interchangeable diagnoses counted as matches.Source: Q14. Please provide details · pp. 26–27

Q20. Please provide any comments on unexpected mappings or other issues.

A.5. Auxiliary Study Objective

  • The auxiliary study recruited 2,052 US adults with a recent health event and provider diagnosis, yielding 1,509 participants with symptoms and a known diagnosis; 445 conversations were sampled for clinician evaluation.A weighted approach prioritized demographic groups underrepresented in the randomized SymptomAI study; combining the auxiliary and SymptomAI datasets produced 2,737 participants with self-reported diagnoses, including 962 assessed by both SymptomAI and baseline clinicians.Source: §A.5 · pp. 28–29

A.6. Auxiliary Study Survey Questions

  • The auxiliary survey covers 39 common primary-care symptom categories and collects an open-response symptom history, preset follow-up answers, a provider diagnosis, and technical and health literacy information.Participants were asked to describe onset, context, symptom qualities, location, severity, duration, and changes over time; the survey history was reformatted as a chat log for evaluation.Source: §A.5 · p. 29, Q12. What type of health insurance do… · p. 32

Q16. Rate the severity of your primary symptoms on a scale of 1–10:

Q7. What is your gender?

Q29. Which internet web search engine did you use?

Q41. I know how to find helpful health resources on the Internet.

A.7. Clinical Evaluation Statistics

The section specifies statistical tests for comparing SymptomAI with clinician baselines, evaluating prompt strategies, and assessing representativeness. Each analysis uses tests suited to its comparison or data type.

Comparing Clinician and SymptomAI DDx

  • Across 1,000 resamples, each case’s SymptomAI DDx was compared with a randomly sampled clinician baseline using exact McNemar tests; median p-values, median odds ratios, and mean Cohen’s g summarized results.Source: §A.7 · p. 36

Clinician Ranking Distribution

  • Clinician selection frequencies for SymptomAI’s first-, second-, or third-best ranking were tested against chance (p=0.333) with exact two-sided binomial tests; preference magnitude used odds ratios and Cohen’s h.Cohen’s h used an arcsine transformation.Source: §A.7 · pp. 36–37

Accuracy by Prompt Strategy

  • Top-5 accuracy for individual and pooled prompt strategies was compared with the Base arm using chi-square tests of independence on independent conversation samples.Source: §A.7 · p. 37

Accuracy Stratified by Conversation Quality & Confidence

  • Within each conversation-quality and confidence stratum, exact McNemar tests assessed whether SymptomAI outperformed baseline clinicians.Source: §A.7 · p. 37

Testing selection bias in participant diagnosis reporting

  • Diagnosis-reporting selection bias was assessed with K-S tests for continuous variables, chi-square tests for categorical variables, and a downsampled, 100-iteration boosted-tree stress test evaluated by mean AUC and its 95% confidence interval.An AUC near 0.50 indicates an absence of systemic multivariate bias.Source: §A.7 · p. 37

Auxiliary Cohort Representativeness

  • Representativeness between the primary clinical-evaluation and auxiliary cohorts was assessed by comparing SymptomAI primary-diagnosis PheCode distributions using a chi-square test and Cramer’s V.Source: §A.7 · p. 37

A.8. Wearable Biosignals Analysis

The analysis linked temporally aligned wearable biosignals to SymptomAI-assigned diagnoses using baseline and acute windows, and separately visualized daily trajectories for acute infections. Skin temperature had no significant associations.

A.8. Wearable Biosignals Analysis

  • The temporal PheWAS fit adjusted logistic regressions for diagnoses with at least 20 cases, comparing Historic (30–4 days before onset) and Recent (3 days before to 3 days after onset) averages.Models adjusted for age, gender, and current weight, used standardized predictors and covariates, and applied a Bonferroni threshold; the window with the lower p-value was designated the primary temporal driver.Source: §A.8 · p. 37
  • Skin temperature showed no significant associations and was excluded; Figure 4 reports odds ratios and 95% confidence intervals for significant biosignal-disease pairs.Nine biosignals were tested, including heart rate, sleep, activity, respiratory, and temperature measures.Source: §A.8 · p. 38
  • For acute infection onset, the analysis visualized temporally aligned daily timeseries for the infectious disease cohort against the remaining population.Source: §A.8 · p. 38

B. Supplemental Experiments · B.1. Performance of LLMs on Existing Diagnosis Benchmarks

Gemini performs well on curated diagnostic benchmarks, but its performance decreases on SymptomAI’s evaluation data, which is more ambiguous and communicated by laypeople.

B.1. Performance of LLMs on Existing Diagnosis Benchmarks

  • Gemini performed well on curated symptom-checker vignettes and complex diagnostic cases, while performance decreased on ambiguous evaluation data communicated by laypeople.The evaluation data included semi-structured surveys with open responses and multiple-choice questions, as well as fully unstructured conversations.Source: §B.1 · p. 38, Table 2 · p. 38

B.2. Auto-rater Consistency Across Studies

The auto-rater aligned strongly with clinician labels and showed consistent ratings of SymptomAI and clinician differential diagnoses across the SymptomAI and auxiliary studies. This supports its use to assess performance across both populations beyond the manually reviewed sample.

B.2. Auto-rater Consistency Across Studies

  • AUC=0.866 and F1=0.918 measured auto-rater alignment with clinicians on 517 clinically rated conversations.Most misalignments occurred when the auto-rater identified a match but clinicians were more conservative; manual review also found ambiguous diagnoses and some clinician-labeling errors.Source: §B.2 · p. 38
  • The auto-rater was applied to diagnosed conversations in the SymptomAI (N=517) and auxiliary (N=445) clinical validation subsamples to compare SymptomAI and baseline clinician DDx.The authors trusted it to assess conversations beyond the clinical evaluation sample and expected verification consistency across both study populations.Source: §B.2 · p. 38
  • Figure 13c shows consistent auto-rated performance for SymptomAI and clinician DDx across the SymptomAI and auxiliary study populations.The authors interpret this consistency as qualitative evidence of generalizability across populations and robustness of auto-rater performance.Source: Figure 13 · p. 39, §B.2 · p. 39

B.3. Auto-rated Performance of Gemini DDx Across Models … E. Prompts

This section describes cross-model and demographic DDx patterns, the illness taxonomy and category-level accuracy analysis, conversation quality, and interview-prompt designs, including prompt variants that differ in when provisional diagnoses are shown and an LLM rubric for matching reported diagnoses to differential candidates.

B.3. Auto-rated Performance of Gemini DDx Across Models

  • Across N=2,737 diagnosed conversations, Gemini variants showed consistent but only slight DDx improvements with recency and size; retrospective interview behavior was not assessed.The manuscript’s accuracy measurements are specific to Gemini 2.0 Flash; subsequently released models were compared using the auto-rater rather than new clinical evaluations.Source: §B.3 · pp. 39–40, Figure 14 · p. 40

B.4. DDx Accuracy Across Demographics

  • DDx performance was mostly consistent across demographics, with subtle advantages among older adults, women, participants with advanced degrees, and those reporting higher online health-resource literacy.Performance did not differ by self-reported medical literacy; the authors suggest greater experience may benefit users, while describing the effects as subtle.Source: §B.4 · pp. 40–41, Figure 15 · p. 41

C. Diagnosed Condition Taxonomy

  • The authors created a two-tier illness taxonomy with 12 parent and 42 granular categories by iteratively coalescing participant-reported diagnoses with Gemini.They used the taxonomy to categorize diagnoses and grouped extracted symptoms for symptom-to-diagnosis Sankey diagrams, omitting temperature dysregulation and fatigue because they occurred across nearly all categories.Source: C. Diagnosed Condition Taxonomy · pp. 41–42

C.1. DDx Accuracy Across Illness Categories

  • Top-1 and top-5 DDx accuracy were assessed across illness categories among participants with self-reported diagnoses from both studies, with results presented in Table 3.The analysis included study arm 1 for posterity; the supplied passages do not provide category-specific accuracy values.Source: §C.1 · p. 42, Table 3 · p. 42

D. Conversation Examples

  • Conversation examples contrast a poor-quality Hashimoto’s example lacking enough information to support the reported diagnosis with a high-quality UTI example where the reported diagnosis was the top DDx candidate.The UTI example elicits symptom timing, frequency, severity, relieving factors, associated symptoms, risk factors, and daily-life impact.Source: D. Conversation Examples · pp. 45–46

E. Prompts

  • The fixed canonical interview instructs the agent to gather detailed HPI information through up to six turns, avoid repeated or irrelevant questions, then provide five reasoned diagnoses and an urgent-care disclaimer.It specifies symptom details, severity, frequency, relieving and worsening factors, medical conditions, risk factors, and functional impact; no more than two questions should be asked per turn.Source: E. Prompts · p. 47
  • The flexible canonical interview likewise gathers HPI details and five reasoned diagnoses within six turns, but adapts questions to relevance and symptom type.It adds associated symptoms and directs the agent not to ask about severity or symptom quality for listed non-pain complaints, or about location when localization is irrational.Source: E. Prompts · p. 48
  • The “Dynamic with Live Updates” prompt provides a best-effort diagnosis and five ranked potential causes during the interview, while asking targeted questions before a final differential.It prioritizes specificity, limits each turn to two questions, and requires a diagnosis list after at most six turns.Source: E. Prompts · p. 49
  • The “Dynamic with Final Output Only” prompt withholds diagnoses during questioning, then produces a first-person symptom history and five reasoned diagnoses with urgent-care guidance.It also directs the agent to use symptom denials, consider less common explanations when details conflict with common diagnoses, and acknowledge uncertainty.Source: E. Prompts · p. 50
  • The rating prompt treats diagnosis matching as a contextual judgment, accounting for abbreviations, combined candidates, typos, lifestyle reports, and clinically meaningful differences in specificity.For example, a specific subtype is not automatically equivalent to a broader diagnosis, while a less-detailed report may match a specific candidate when the conversation supplies the context.Source: E. Prompts · p. 51
  • The rubric generally matches suspected diagnoses to candidates, distinguishes partial matches from exact ones, and requires contextual support before matching a parent condition to a specific subtype.It requests raw XML with a match position (or -1), reasoning, and an ambiguity score from 1 to 5.Source: E. Prompts · p. 51

Related papers

Loading 2605.04012v2…