Source-linked AI summary
CareGraph: An Auditable Hybrid AI Framework for Evidence-Grounded Personalized Longitudinal Health Intelligence
Pratik Ghawate, Tanvi Patil
TL;DR
Fragmented longitudinal health records are difficult to interpret and trace while avoiding unsupported or autonomous clinical decisions. CareGraph addresses this with deterministic evidence preparation, provenance-linked constrained synthesis, safety validation, and release gating. In controlled synthetic evaluations, it achieved strong component results and operationally governed outputs, while its comparison with a monolithic baseline exposed a breadth-versus-traceability trade-off.
Problem
Fragmented clinical, self-reported, and wearable evidence makes longitudinal change, missing context, discussion priorities, and provenance difficult to interpret.
Method
CareGraph integrates heterogeneous records through deterministic analysis, authorized evidence registries, graph provenance, constrained synthesis, safety controls, and release gating.
Results
CareGraph achieved strong held-out trend, missing-context, and safety results, while the 80-patient audit and 56-patient comparison showed governed completion and a focused, traceable trade-off against a broader monolithic baseline.
Takeaways & Limitations
CareGraph provides a safety-bounded backbone for evidence-governed longitudinal health intelligence with inspectable provenance and fail-closed release boundaries.
Takeaways & Limitations
Evaluation was controlled and synthetic, with no real clinical workflow or patient-outcome assessment; scenario conditioning also creates a potential leakage channel.
Abstract
from arXiv · showhide
Artificial intelligence is transforming personalized healthcare, yet fragmented clinical, self reported, and wearable evidence remains difficult to interpret and trace. We present CareGraph, an auditable hybrid AI framework that converts heterogeneous records into prioritized trends, missing context indicators, bounded next steps, discussion questions, and provenance linked explanations. CareGraph organizes evidence without diagnosing, predicting outcomes, selecting treatment, or making autonomous clinical decisions. Its pipeline covers deterministic analysis, context detection, graph construction, constrained language model synthesis, evidence validation, safety controls, and release gating. Tests used synthetic cohorts of 400 patients each for development, validation, and holdout. On holdout data, a frozen ordinary least squares trend rule with a sufficiency gate achieved 0.827 accuracy, 0.837 macro F1 with a 95 percent confidence interval of 0.819 to 0.854, and 0.974 insufficient data F1. Missing context detection achieved 0.815 strict micro F1 versus 0.318 for the legacy detector. On an authored holdout benchmark, safety ruleset version 1.2 achieved 1.000 precision, 0.950 recall, and 0.974 F1. An audit requiring graph retrieval across 80 patients yielded 79 syntheses and 78 presentations without fallback; one output was blocked and one failed closed because of an invalid evidence key. Against monolithic GPT 5.6 on 56 matched patients, CareGraph was faster at 40.15 versus 49.62 seconds, shorter at 661 versus 1,163 words, and showed better exploratory lexical alignment with longitudinal targets; the baseline used fewer tokens and cited more raw evidence. Graph auditing verified provenance and deterministic retrieval; incremental graph effects on generation require paired evaluation. CareGraph offers a safety bounded foundation for intelligent personalized health systems.
1 Introduction
CareGraph treats personalized longitudinal health intelligence as an evidence-organization and governance problem created by fragmented records. It combines deterministic analysis, provenance, constrained synthesis, safety review, and release controls without autonomous clinical decision-making.
- Fragmented laboratory, clinical, symptom, lifestyle, history, and wearable records obscure longitudinal change, missing context, discussion priorities, and provenance.
- CareGraph separates deterministic evidence generation, constrained language-model synthesis, safety validation, and release governance.
- The framework is extensible: new data adapters can populate a shared patient state while downstream trend, provenance, synthesis, and release contracts remain stable.
- The evaluation spans frozen component cohorts, safety benchmarks, graph provenance audits, operational reliability, and matched comparison with a monolithic baseline.
- CareGraph does not train a clinical prediction model; it focuses on transparent patient-level evidence preparation and bounded synthesis.
3 Problem Definition and Research Questions
CareGraph formalizes personalized longitudinal health intelligence as mapping heterogeneous patient records into structured, evidence-linked outputs. Its research questions evaluate component performance, provenance, operational behavior, safety, and comparison with a monolithic baseline.
- For patient p, heterogeneous longitudinal records are mapped into a structured output containing summaries, findings, bounded next steps, clinician questions, uncertainty, and evidence references.
- Every generated finding or next step must have nonempty evidence drawn from the authorized patient registry Kp.
- Existing evidence-key validity is necessary but insufficient because generated text must also accurately represent the cited evidence.
- The study asks whether frozen trend classification separates data sufficiency from direction, how schema-aligned missing-context detection compares with legacy detection, and how accurately safety rules detect adversarial language.
4 System Architecture
CareGraph places deterministic evidence and explicit context handling before constrained synthesis, then separates probabilistic generation from machine-checkable validation, safety, and release.
- The architecture combines source-extensible adapters, deterministic sufficiency and trend agents, missing-context handling, a patient-scoped provenance graph, constrained synthesis, and release gates.
- Every generated finding and action must cite at least one key in the authorized registry Kp, which aggregates deterministic and retrieved patient evidence.
- Presentation is allowed only after schema, patient identity, evidence, safety, and required-artefact contracts pass; otherwise outputs fail closed.
- The pipeline ranks evidence deterministically, freezes model and input metadata, rejects invalid registries or keys, applies safety rules, and archives outputs and failure metadata.
- Generation is probabilistic, whereas admissible evidence, schema validation, failure handling, safety transformation, and release are machine-checkable.
5 Data and Experimental Protocol
The study uses independently seeded synthetic cohorts and frozen evaluation artefacts to assess trends, missing context, safety, provenance, and operational behavior under controlled conditions.
- Development, validation, and replacement held-out cohorts each contain 400 independently generated patients across eight scenarios, with three annual encounters and 1,700 expected trend labels.
- The benchmark covers metabolic, cardiovascular, kidney, liver, anaemia, thyroid, medication-safety, and inflammation or autoimmune scenarios.
- Scenario information is supplied directly to runtime inventory and missing-context rules, creating a benchmark limitation and potential leakage channel.
- Source hashes, cohort manifests, runtime checks, required artefacts, and evaluation packages were frozen or verified before analysis.
- Metrics include classification, missing-context, safety, bootstrap confidence intervals, and paired or corrected statistical tests.
6 Component Methods
CareGraph combines sufficiency-gated trend analysis, context detection, graph-based provenance, constrained synthesis, and safety controls. Its evaluation also identifies important boundaries: benchmark labels have limited semantic grounding, graph-generation effects remain untested, and the broad comparison design was incomplete.
- Trend Agent: A sufficient series requires at least three valid measurements spanning 365 days, with comparable units, parseable values, no unresolved conflicts, and nonzero scale.Failed gates emit explicit reasons and map to insufficient_data for benchmark scoring.
- Trend Agent: The selected OLS score multiplies fitted slope by observed span and normalizes by the magnitude of the first value.Increasing, decreasing, and stable labels are assigned using thresholds; OLS with τ = 0.08 was frozen after development and validation comparisons.
- Evaluation Boundaries: Strict benchmark scores may reflect agreement with generator rules rather than independent semantic truth because contradiction states, incidental missingness, and some source-observable labels were limited.The audit notes that apparent false positives can be plausible omissions and false negatives can reflect rule gaps.
- Patient Graph: The patient graph represents heterogeneous evidence and typed provenance links, with deterministic ranking based on scenario, relationship, contradiction, temporal, trend, medication, and symptom relevance.Nodes include patient data, trends, missing context, questions, and multiple clinical and wearable evidence types.
- Patient Graph: The graph audit measures loading, retrieval, source-row recovery, orphan edges, and provenance completeness, while generation effects are reserved for paired graph/no-graph evaluation.The planned paired evaluation holds other variables constant to assess effects on findings, omissions, unsupported claims, and reviewer preference.
- Structured Synthesis: Constrained synthesis receives a patient-scoped insight object, ranked graph evidence, and authorized registry keys rather than unrestricted raw tables.Generated findings and next steps must cite authorized evidence keys, with extra fields forbidden and AI and graph inputs optionally required.
- Paired Baseline: The broad comparison did not execute its complete 1,200-run design, using instead 56 matched patients sharing the frozen final prompt hash.Metabolic-risk and cardiovascular-risk cases generated under earlier prompt hashes were excluded from the matched analysis.
7 Results
Held-out component tests supported CareGraph’s deterministic trend, missing-context, and safety stages, while audits verified provenance and release behavior. The matched comparison found faster, shorter CareGraph outputs with stronger exploratory alignment, but graph-specific generation effects and clinical relevance remain untested.
- 7.1 Held-Out Trend Classification: 0.827 accuracy and 0.837 macro-F1 were achieved by the frozen OLS trend implementation on 1,700 held-out labels.The explicit sufficiency gate achieved insufficient-data F1 0.974; the weakest scenario was inflammation/autoimmune signal and ANA was the weakest biomarker.
- 7.2 Missing-Context Detection: 0.815 strict micro-F1 was achieved by version 2 for missing-context detection, with a paired mean patient-level set-F1 improvement of 0.452 over legacy.Version 2 had precision 0.708 and recall 0.961 across 400 paired patients.
- 7.3 Safety Rule Evaluation: 1.000 precision, 0.950 recall, and 0.974 F1 were achieved by safety ruleset v1.2, with a 0.000 false-positive rate.One treatment-planning paraphrase remained a false negative in the authored holdout benchmark.
- 7.5 80-Patient Operational Audit and Evidence Status: 79 of 80 scheduled patients completed synthesis and 78 completed presentation without fallback; one synthesis was blocked and one failed closed on an invalid evidence key.Across successful syntheses, no invalid source keys remained and no priority finding or next step lacked an evidence key.
- 7.5 80-Patient Operational Audit and Evidence Status: 56 findings or steps were removed across 44 patients during operational safety review, with inspection indicating apparent false positives around medication context.This operational finding was retained as a safety-generalisation limitation rather than combined with the small authored benchmark.
- 7.6 Matched 56-Patient Monolithic Comparison: CareGraph was 9.46 seconds faster and produced 502 fewer words than the monolithic baseline, while using 3,994 more total tokens.The baseline produced more findings and evidence references on average; these measures assess generation behavior and traceability breadth, not clinical correctness.
- 7.6 Matched 56-Patient Monolithic Comparison: 93.9% versus 53.7% trend-direction alignment favored CareGraph over the monolithic baseline in exploratory lexical checks.Missing-context term alignment was 97.0% versus 67.7%, but the checks are sensitive to wording and benchmark design.
- 7.6 Matched 56-Patient Monolithic Comparison: No final-prompt CareGraph output cited graph-evidence keys, so incremental graph effects on generation were not demonstrated.Graph retrieval and provenance were verified, but paired graph/no-graph generation and review did not run.
8 Discussion
CareGraph’s modular architecture separates evidence calculation, provenance, constrained synthesis, and release control to produce focused, inspectable longitudinal health intelligence. Results support its alignment within the predefined synthetic objective, while graph reasoning itself remains unproven.
- Architectural contribution: CareGraph separates patient-state integration, deterministic analysis, graph provenance, constrained language generation, safety review, and presentation gating.The architecture organizes only authorized evidence for synthesis and controls whether outputs can be presented.
- Controlled evidence for core components: 0.974 insufficient-data F1 shows that the data-sufficiency gate was more consequential than the small difference between endpoint- and slope-based trend candidates.The same component evaluation reported missing-context micro-F1 improvement from 0.318 to 0.815 after schema alignment.
- Graph contribution: 0.979 mean provenance completeness, zero orphan edges, and complete source-row recoverability establish the graph as a durable provenance substrate.The study did not demonstrate a separate graph-reasoning advantage, and deterministic summaries often supplied the concise evidence used for generation.
- System-level comparison: CareGraph produced faster, shorter outputs more consistently aligned with predefined longitudinal targets, whereas the monolithic baseline used fewer tokens, cited more raw evidence, and surfaced broader secondary context.The comparison frames a trade-off between focused, governable synthesis and broader observation coverage.
- Research implications: Within the predefined synthetic longitudinal-synthesis objective, CareGraph is the better-aligned architecture rather than evidence of more universal medical knowledge than a monolithic LLM.Its supported contribution is a focused and governable backbone with an inspectable path to the underlying record.
9 Study Scope and Threats to Validity
The study’s validity is bounded by controlled engineering comparisons and synthetic or nonclinical evaluation. Clinical relevance, workflow effects, and broader generalization therefore remain unresolved.
- Threats to validity: Table 9 identifies the study’s principal threats to validity, but the supplied passage provides no threat details beyond the table title.
- Evaluation boundaries: Clinical relevance, important omissions, prioritization, usefulness, and preference were not evaluated in the current engineering comparison.A blinded physician-review packet was prepared for a separate phase.
- Evaluation boundaries: No real clinical workflow or patient-outcome evaluation was performed, so claims are limited to controlled synthetic performance and system behavior.
10 Ethical, Privacy, and Safety Considerations
CareGraph is scoped to organizing health information and preparing clinician discussion, not making clinical decisions. Synthetic data and untrusted documents introduce privacy, fairness, interoperability, and security boundaries requiring further governance.
- Privacy and data limitations: Synthetic cohorts avoid identified real-patient data but may encode unrealistic assumptions, demographic simplifications, and hidden fairness failures.Future real-data work would require governance, access controls, encryption, audit logging, retention policies, incident response, and regulatory assessment.
- Intended use and safety: CareGraph organizes longitudinal health information and discussion preparation without diagnosing, prescribing, altering medication, triaging emergencies, or replacing professional judgment.
- Intended use and safety: The safety rules reduce a bounded set of surface-language risks but are not a safety guarantee.
- Interoperability and security: FHIR and controlled terminologies may improve interoperability, but their standard mapping must be evaluated rather than assumed.Untrusted documents also create prompt-injection, parser, unit-corruption, and fabricated-record risks requiring separately governed stages.
11 Translation Roadmap and Next-Phase Evaluation
The next phase prioritizes immutable reliability evaluation, clinician adjudication, graph-specific paired testing, governed data expansion, and prospective workflow assessment. These steps target the current study’s operational, clinical, semantic, interoperability, and reproducibility gaps.
- Reliability and release engineering: Freezing one uniquely versioned synthesis prompt and corrected safety rules will enable a repeat end-to-end cohort estimate without sequential prompt drift.The planned archive includes response identifiers, runtime, token use, costs, hashes, and failure metadata.
- Clinical evaluation: A blinded physician-review study will independently score relevance, important omissions, factual support, prioritization, safety, usefulness, and pairwise preference.
- Clinical evaluation: A second clinician on a stratified subset would add inter-rater agreement and convert the engineering comparison into a clinically adjudicated evaluation.
- Graph contribution and semantic evidence: A paired graph/no-graph generation study with identical settings would isolate the graph’s incremental value and distinguish valid key membership from faithful evidence use.
- Interoperability and data expansion: Governed adapters should extend the backbone to FHIR-based EHRs, laboratory portals, medication histories, patient-reported outcomes, and wearables through the same evidence contracts.
- Prospective translation and reproducibility: After governance, security, and intended-use review, real-workflow evaluation should measure comprehension, trust, clinician-discussion quality, and downstream action.Reproducibility requires tagged releases, licenses, frozen manifests, and archival snapshots mapping results to immutable experimental states.
12 Conclusion
CareGraph combines analytical focus, inspectable provenance, and fail-closed release controls for longitudinal health-information synthesis. Within the controlled study, it remains a research framework for information organization and clinician-discussion preparation rather than diagnosis or autonomous clinical decision-making.
- 0.827 held-out accuracy and 0.837 macro-F1 were achieved by the frozen trend implementation, including 0.974 insufficient-data F1.These results were reported under independently seeded synthetic evaluation.
- 0.815 missing-context micro-F1 exceeded 0.318 for the legacy detector.
- 79 of 80 graph-required syntheses completed and 78 reached presentation without fallback; one output was blocked and one stopped before release after an invalid evidence key.The stress test demonstrated operational completion alongside failure-aware governance.
- CareGraph combines analytical focus with inspectable evidence and fail-closed release boundaries for evidence-governed longitudinal synthesis.Its architecture is designed to expand through new data adapters and clinical evaluation without redesigning the core pipeline.
- The framework is bounded to health-information organization and clinician-discussion preparation, not diagnosis, treatment recommendation, autonomous clinical decision-making, or a clinically validated medical device.
- The reported study used entirely synthetic records and included neither human participants nor identifiable private information.Formal institutional review was not obtained because the analysis included no human-subject data or clinician ratings.