Source-linked AI summary
Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Yuan Pu, Yewon Chang, Furong Jia, Xunjian Yin, Jessica Ma, Ayman Ali, Monica Agrawal
TL;DR
Health chatbot evaluations often rely on APIs, although consumer interfaces can differ from APIs even when model versions and settings are matched. This study compares health responses across access modes and finds systematic differences, especially in response form and engagement behavior, supporting the need for faithful replication of consumer settings in audits.
Problem
Most evaluations use APIs although chatbot interfaces and APIs can produce different responses, and health-specific ChatGPT features remain largely untested against API access.
Source: §1 · p. 2Method
The study compares repeated responses to the same health questions across GPT versions and available access modes, assessing response form, behavior, sources, and clinical content.
Source: §3 · p. 4, Response form · p. 4, Response behavior · p. 4Results
The largest and most consistent access-mode divergences were in readability, formatting, and engagement behavior, and some differences reversed direction across GPT versions.
Source: §4 · p. 6Takeaways & Limitations
Valid audits require a scalable way to recreate the settings through which consumers encounter chatbots, which are not publicly documented or accessible in sufficient detail for independent evaluation.
Source: §4 · p. 6The study tested unusually controlled, single-turn conditions without memory, connected records, conversation history, or user customization; routine-use features could further widen the gap.
Source: §4 · p. 6
Abstract
from arXiv · showhide
People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying settings. Here, we find systematic differences across access modes. Because evaluations typically rely on APIs while consumers interact through chatbot interfaces, these discrepancies limit evaluation validity. Our findings underscore an urgent need for model providers to enable faithful replication of consumer experiences and settings for rigorous audits.
1 Background and Motivation
Health advice is increasingly obtained through consumer chatbots, while most evaluations use APIs; the two access systems need not produce equivalent responses. Health adds features that APIs cannot reproduce, but its differences from API responses to the same health questions remain largely untested.
1 Background and Motivation
- Most LLM health evaluations use APIs for scalable, high-throughput access, while chatbot-based collection mirrors consumer use but is difficult to scale amid rapid model updates.Evaluations may generalize findings from either mode as though they characterize a single system.Source: §1 · p. 2
- Consumer interfaces and APIs may differ in instructions, context, tools, and safeguards even with the same model version and matched settings, and many interface configurations are not publicly documented.Researchers can configure analogous API settings but cannot fully replicate the consumer-facing environment.Source: §1 · p. 2
- Empirical comparisons of chatbot and API access remain limited in healthcare, despite reports of output differences in other settings.Prior non-health comparisons describe differences in source selection, source diversity, and behavior in disordered or conspiratorial dialogue [16].Source: §1 · p. 2
- ChatGPT Health adds health-specific training, automatic activation for detected personal health questions, and optional use of connected medical records and wellness data.These health-specific features cannot be reproduced through the API.Source: §1 · p. 2
2 Study Design
The study compares responses to the same health questions across available model versions and access modes, using fresh sessions without connected data or prior personalization. It assesses differences in response form, behavior, sources, and clinical content.
2 Study Design
- The study compares responses to the same consumer health questions across GPT-5.3 and GPT-5.4 and their available access modes, using fresh sessions without connected data, conversation history, memory, or user customizations.Source: §2 · p. 2
- Figure 1 conceptually contrasts researcher-controlled API, ChatGPT, and ChatGPT Health access, emphasizing that surrounding instructions, context, tools, and safeguards may differ.The schematic is conceptual and does not depict OpenAI systems’ internal architecture.Source: Figure 1 · p. 3
- For GPT-5.3, the study evaluates standard ChatGPT, ChatGPT Health, and API access; for GPT-5.4, it compares standard ChatGPT with the API, with Extended Thinking and web search enabled.Source: §2 · p. 3
- Each of 50 patient questions from an online medical consultation platform was submitted three times in each version-mode condition, with paired within-question comparisons across response dimensions.The compared dimensions included response form, behavior, referenced web sources for GPT-5.4, and clinical content.Source: §2 · pp. 3–4
3 Results
Responses differed systematically across access modes, even for the same health questions and model version. The largest and most consistent divergences concerned response form and engagement behavior, while source selection and clinical content also varied.
3 Results
- Differences across access modes were systematic across the 50 questions studied, including 42 questions for comparisons involving ChatGPT Health.Figure 2 illustrates differences between GPT-5.3 API and ChatGPT Health responses to the same question.Source: §3 · p. 4, Figure 2 · p. 5
Response form.
- GPT-5.3 interface responses were longer, easier to read, and more densely formatted than API responses, whereas GPT-5.4 interface responses were longer but harder to read and less densely formatted.All stated GPT-5.3 differences and the GPT-5.4 readability and formatting-density differences had p < 0.001; GPT-5.4 length difference had p = 0.003. GPT-5.4 responses from neither mode used emojis.Source: Response form · p. 4
Response behavior.
- GPT-5.3 ChatGPT and ChatGPT Health responses invited further interaction more often than API responses, but GPT-5.4 API responses more often made continuation moves than ChatGPT responses.Self-limiting statements were uncommon across GPT-5.3 modes and more common in GPT-5.4, particularly through the API.Source: Response behavior · p. 4
Referenced sources.
- 10.9% webpage and 29.8% domain Jaccard overlap across GPT-5.4 modes was lower than within-mode overlap, which averaged about 17–18% for webpages and 39% for domains.The cross-mode differences versus within-mode overlap had p < 0.001; modes cited similar numbers of webpages and domains but often different ones.Source: Referenced sources · p. 4
Clinical content.
- Clinical concepts and named levels of care varied less by access mode than response features, while additional-concept overlap was lower across than within modes in several comparisons.For GPT-5.3 additional concepts, ChatGPT versus API overlap was 0.615 vs. 0.653 (p = 0.012), and Health versus API was 0.614 vs. 0.662 (p = 0.002); Health versus standard ChatGPT was 0.645 vs. 0.654 (p = 0.617). GPT-5.4 ChatGPT versus API overlap was 0.586 vs. 0.619 (p = 0.013).Source: Clinical content · p. 5
4 Discussion
Access-mode differences persisted under controlled conditions, with the largest and most consistent divergences in readability, formatting, and engagement behavior. Because consumer configurations are not sufficiently documented or accessible for independent replication, the authors argue that rigorous audits require scalable access to consumer deployment pathways.
4 Discussion
- The largest and most consistent access-mode divergences were in readability, formatting, and engagement behavior, which may shape patient comprehension and health outcomes.Some differences also reversed direction across GPT versions, so access-mode effects were not stable across versions.Source: §4 · p. 6
- Consumer configurations are not publicly documented or accessible in sufficient detail for independent evaluation, limiting faithful replication of chatbot experiences [20].Source: §4 · p. 6
- Even under controlled, single-turn conditions without memory, connected records, history, or customization, the study found significant access-mode differences.Routine consumer features may further widen the gap; the authors cite an alleged case involving religiously customized health advice and a pulmonary embolism.Source: §4 · p. 6
- The authors frame inability to replicate consumer experiences as a regulatory blind spot that researchers cannot fix on their own.Source: §4 · pp. 6–7
- The authors argue that developers should provide researchers scalable, reproducible access to consumer deployment pathways, including major features such as EHR connections.Without this access, independent evaluation risks measuring the model available to auditors rather than the product patients experience.Source: §4 · p. 7
5 Methods
The study compared repeated responses to patient-authored health questions across GPT versions and consumer access modes, then analyzed response features using paired, within-question comparisons. Its methods also account for run-to-run variation and distinguish within-mode from cross-mode consistency for set-valued features.
5.1 Consumer health question collection
- Fifty patient-authored questions were sampled after excluding explicit clinician-directed queries, and only the patient question text was given to models.Physician answers and other response-side information were withheld.Source: §5.1 · p. 8
5.2 LLM response collection
- The primary collection produced 600 responses across two GPT versions, two access modes, 50 questions, and three runs; Health added 126 responses for 42 eligible questions.Repeated runs were used to characterize response variability under the same nominal condition.Source: §5.2 · p. 8
ChatGPT Interface.
- Interface runs used new Temporary Chats; GPT-5.4 outputs included the rendered pre-answer preamble, which the copy function did not export.Source: ChatGPT Interface · p. 8
ChatGPT Health.
- Health was manually selected for all 50 questions but activated for 42; no records or health applications were connected, and API prompts and settings were chosen to mirror interface behavior.The remaining eight questions concerned another person’s health and did not activate Health; Health runs used GPT-5.3 Instant with web search disabled.Source: ChatGPT Health · p. 9
5.3 LLM response processing
- Processing removed Markdown for length and readability measures but retained formatting features and separated GPT-5.4 web references from response prose.The primary GPT-5.4 analysis included the user-visible preamble; final-answer-only length analyses were reported in the Appendix.Source: §5.3 · p. 9
5.4 Response features
- Response features covered length, readability, formatting, self-limiting language, continuation moves, referenced sources, added clinical concepts, and named levels of care.Self-limiting language was narrowly defined using “I can’t” or “I cannot”; clinical concepts were extracted with a GPT-5.4-based pipeline and clinician-reviewed prompts.Source: §5.4 · pp. 9–10
5.5 Statistical analysis
- Comparisons were paired within question and model version; scalar and binary features used Wilcoxon tests, while set-valued features used question-level Jaccard overlaps.Cross-mode set overlap was compared with the mean of the two within-mode overlaps; empty unions were excluded, so analytic sample sizes varied by feature and mode pair.Source: §5.5 · pp. 10–11
Appendix A Additional Tables
The appendix tables describe response features and paired access-mode contrasts, alongside within- and cross-mode Jaccard overlap measures. Reported contrasts include differences in offers to produce more, readability, and additional clinical concepts.
Appendix A Additional Tables
- Table A1 summarizes response features and within-mode repeated-run overlap, while Table A3 defines within- versus cross-mode Jaccard contrasts for each access-mode pair.Table A1 includes 50 questions for API and standard ChatGPT and 42 for ChatGPT Health; Table A3 reports within minus cross as its difference.Source: Table A1 · p. 15, Table A3 · p. 17
- Table A2 reports a Flesch–Kincaid grade paired difference of −0.89 (p<0.001), but the supplied excerpt does not identify its access-mode contrast; lower grades indicate easier readability.Source: Appendix A · p. 16, Table A2
- For additional clinical concepts, the excerpt reports paired-difference cells +0.73 (p<0.001), +0.09 (p=0.900), −0.56 (p<0.001), and −0.73 (p=0.005), but their mode contrasts cannot be identified.Contrasts involving ChatGPT Health use the matched 42-question subset.Source: Appendix A · p. 16, Table A2