Source-linked AI summary
BEACON-SP: Ontology-Grounded GraphRAG Framework for Clinical Suicide Risk Assessment
Kemal Davaslioglu, Nathan Conger, Sastry Kompella, Yalin E. Sagduyu, Nathaniel D. Bastian
TL;DR
Suicide-risk assessment requires integrating heterogeneous clinical, behavioral, social, and temporal evidence, but such synthesis remains challenging. BEACON-SP combines ontology-grounded patient graphs with GraphRAG for clinician-facing decision support. On a 1,500-query benchmark, GraphRAG was preferred in 76.4% of head-to-head comparisons, with gains in clinical relevance and completeness.
Problem
Integrating interacting clinical, behavioral, social, and temporal factors into actionable suicide-risk assessments remains challenging.
Source: §I · p. 1Method
Results
GraphRAG was preferred in 76.4% of 5,864 head-to-head criterion evaluations, with clinical relevance at 93.9% and completeness at 84.0%.
Source: D. Discussion · p. 5Takeaways & Limitations
BEACON-SP is intended to retrieve and contextualize patient evidence for clinician support, not to assess risk autonomously or replace clinician judgment.
Source: §I · p. 1RAG outperforms GraphRAG on suicide indicators, while both systems remain below 0.5 on protective-versus-risk balance, ontology mapping, and summary tasks.
Source: D. Discussion · p. 5
Abstract
from arXiv · showhide
We present BEACON-SP, an ontology-grounded Graph Retrieval-Augmented Generation (GraphRAG) framework for clinician-facing decision support in behavioral health settings such as suicide prevention, where effective assessment requires integrating heterogeneous clinical, behavioral, social, and temporal evidence. BEACON-SP combines patient knowledge graphs with ontology-guided retrieval to support multi-hop reasoning across diagnoses, medications, risk and protective factors, life events, and temporal relationships. The framework is enabled by a comprehensive suicide prevention ontology that integrates the Three-Step Theory, the Integrated Motivational-Volitional Model, and the Suicide Social Determinants of Health Ontology into a unified representation of patient risk factors. We construct ontology-grounded patient knowledge graphs and evaluate BEACON-SP for clinician-facing question answering. Compared with a vector-based retrieval-augmented generation (RAG) baseline on a 1,500-query benchmark spanning 15 clinical categories and 100 patients, BEACON-SP improves completeness, clinical relevance, and evidence grounding under a corrected comparative evaluation protocol, with a small gain on factual accuracy. In paired criterion-level comparisons, GraphRAG is preferred in 76.4% of cases. These results demonstrate the potential of ontology-guided GraphRAG to provide structured, contextualized patient evidence for clinical decision support.
I. INTRODUCTION
Suicide-risk assessment requires integrating interacting psychiatric, social, environmental, and temporal factors, yet existing tools do not comprehensively combine diagnoses, social determinants, and behavioral indicators. BEACON-SP proposes an ontology-grounded GraphRAG framework and a reproducible patient-graph pipeline, alongside a comparative evaluation protocol designed to address bias against graph systems.
I. INTRODUCTION
- Existing clinical tools do not comprehensively integrate diagnoses, social determinants, and behavioral indicators into a unified suicide-risk assessment.The paper links this gap to limited support for timely, personalized intervention and evaluation of veteran suicide risk.Source: §I · p. 1
- BEACON-SP combines patient-specific knowledge graphs, ontology-guided retrieval, and graph-based evidence expansion to support multi-hop reasoning for clinician-facing suicide-risk decision support.It is designed to contextualize evidence and support structured risk formulation, not to assess risk autonomously or replace clinician judgment.Source: §I · p. 1
- The framework’s suicide-prevention ontology integrates the Three-Step Theory, the Integrated Motivational-Volitional model, and the Suicide SDoH Ontology into a unified patient-level representation.The representation covers diagnoses, medications, risk and protective factors, life events, and temporal relationships.Source: §I · p. 1
- The paper presents a reproducible pipeline for constructing ontology-grounded patient knowledge graphs from electronic health records.Source: §I · p. 1
- The authors identify a systematic evaluation bias against graph-based retrieval and introduce a protocol that evaluates each system against its own retrieved evidence.They also evaluate BEACON-SP against vector-based RAG across 1,500 queries, 15 suicide-prevention tasks, and 100 patients.Source: §I · p. 1
II. RELATED WORK
Prior healthcare and mental-health work has applied RAG, knowledge graphs, and LLM assistants to clinical tasks. BEACON-SP distinguishes itself through a unified ontology-grounded patient graph for theory-informed relational and temporal reasoning.
II. RELATED WORK
- Prior work applies GraphRAG to medical literature and EHRs, clinical risk prediction, health-claim fact-checking, depression detection, and multimodal healthcare assistance.Source: §II · p. 2
- BEACON-SP extends prior contextual RAG work with ontology-grounded patient graphs designed for relational and temporal reasoning.Source: §II · p. 2
- Unlike approaches based on similar text chunks or generic mental-health guidance, BEACON-SP unifies 3ST, IMV, and Suicide-SDoHO in a patient graph that supports single- and multi-hop questions.Source: §II · p. 2
III. SUICIDE RISK MODELING
BEACON-SP draws on three complementary frameworks to represent patient-level suicide risk. Together, they connect clinical observations with theory-grounded risk constructs and temporal context.
III. SUICIDE RISK MODELING
- Three-Step Theory characterizes risk through psychological pain, hopelessness, connectedness, and capacity for suicide.Ideation emerges when pain and hopelessness co-occur; connectedness affects its severity, while capacity factors relate to progression to attempt.Source: §III · p. 2
- The Integrated Motivational-Volitional model organizes suicide risk into pre-motivational, motivational, and volitional phases.These phases cover background vulnerability, ideation development through defeat and entrapment, and transition from intent to behavior.Source: §III · p. 2
- Suicide-SDoH provides a vocabulary for social and contextual contributors, and the three frameworks together link clinical observations to risk constructs and temporal context.Source: §III · p. 2
IV. ONTOLOGY FRAMEWORK
The BEACON-SP ontology links multiple source ontologies through a core and bridge layer to represent patient features. Its domains cover suicide-risk factors, clinical information, and temporal processes and events.
IV. ONTOLOGY FRAMEWORK
- The BEACON-SP ontology contains 247 attributes across 12 feature domains, with the BEACON Core and Bridge connecting source ontologies to patient features.The architecture includes 3ST, Suicide-SDoH, PACO, and TEO.Source: §IV · p. 2
- Risk factors are represented across ten Suicide-SDoH categories, while protective factors include connectedness, coping skills, stable employment, means restriction, and effective clinical care.Source: §IV · p. 2
- The ontology represents temporal information through state transitions, clinical epochs, observation series, and bounded risk windows triggered by events.Examples of risk-window triggers include post-discharge and medication initiation.Source: §IV · p. 3
V. PATIENT KNOWLEDGE GRAPH AND GRAPHRAG IMPLEMENTATION
BEACON-SP addresses the limits of flat chunk retrieval by representing patient evidence and its relationships in an ontology-grounded graph. Semantic seed retrieval followed by bounded graph traversal supplies structured relational and temporal context for clinical questions.
V. PATIENT KNOWLEDGE GRAPH AND GRAPHRAG IMPLEMENTATION
- Vector-based RAG retrieves semantically similar chunks, but can miss clinically related evidence distributed across diagnoses, medications, social history, life events, and longitudinal transitions.Its top-k chunks are concatenated into the language-model context.Source: §V · p. 3
- BEACON-SP maps patient attributes and derived constructs into typed graph nodes and edges that encode clinical, temporal, and ontology relationships.The ontology supplies the schema; theory constructs and temporal features become higher-order nodes.Source: §V · p. 3
- The retrieved patient-specific subgraph is passed to the language model as structured context.Patient-specific identifiers are used to prevent cross-patient contamination.Source: §V · p. 3
- At query time, cosine-similarity matching selects patient-specific seed nodes, then bounded breadth-first expansion retrieves neighboring nodes and relations.Expansion is constrained by patient identifier, edge type, and traversal depth; the resulting subgraph is serialized with labels, timestamps, and source attributes.Source: §V · p. 3
A. Experimental Setup
The evaluation uses synthetic Veterans Affairs suicide-prevention records and compares GraphRAG with vector-based RAG while holding answer generation and prompts constant. The benchmark spans 1,500 queries, 100 patients, and 15 suicide-prevention categories.
A. Experimental Setup
- The study uses Veterans Affairs Suicide Prevention Synthetic Dataset records to construct the BEACON-SP knowledge graph and conducts qualitative and quantitative assessments.The qualitative assessment compares both retrieval systems on the same query.Source: A. Experimental Setup · p. 3
- The quantitative benchmark covers 1,500 clinical queries from 100 patients across 15 suicide-prevention categories.Both systems share the answer-generation model and prompting, differing in retrieval mechanism.Source: A. Experimental Setup · p. 3
- Vector-based RAG retrieves similar patient-record chunks, while GraphRAG uses cosine-similarity seeds followed by breadth-first expansion over patient-specific graph nodes and relations.The shared generation model and prompts isolate retrieval structure as the difference between systems.Source: A. Experimental Setup · pp. 3–4
B. Qualitative Evaluations
A representative multi-hop psychiatric-history query shows GraphRAG organizing diagnostic events into a longitudinal chronology, while vector-based RAG returns no relevant information for the same question. The example illustrates the value of explicit temporal and diagnostic relations for reconstructing clinical trajectories.
B. Qualitative Evaluations
- GraphRAG reconstructs a dated diagnostic chronology from 1993 through 2021, including onset, psychotic features, schizophrenia, anxiety, and a relapsing course without documented remission.It summarizes the trajectory through chronicity, comorbidity, and ongoing pharmacologic management.Source: B. Qualitative Evaluations · p. 4
- For the same psychiatric-history query, vector-based RAG returns no relevant information despite the underlying record containing the relevant facts.The query concerns a high-risk patient with PTSD, GAD, schizophrenia, and chronic pain.Source: B. Qualitative Evaluations · p. 4
- Reconstructing the trajectory requires linking diagnoses across time, distinguishing onset from relapse, and tracing accumulating comorbidity—relations encoded in the graph but not assembled by flat chunk retrieval.The example contrasts graph traversal with retrieval of locally similar text.Source: B. Qualitative Evaluations · p. 4
C. Quantitative Evaluations
Under comparative evaluation, GraphRAG is preferred overall and leads on clinical relevance, completeness, and evidence grounding, while factual accuracy shows only a small advantage. Absolute scoring against flat patient records can penalize graph-derived constructs, so the paper uses each system’s retrieved evidence for comparative verification.
C. Quantitative Evaluations
- The four evaluation criteria are factual accuracy, completeness, clinical relevance, and evidence grounding; the comparative judge sees both answers and their respective retrieved evidence.Responses are scored on a trinary scale of 0, 0.5, or 1.Source: C. Quantitative Evaluations · p. 4
- Comparative judging checks each answer against its retrieved evidence, mitigating flat-record penalties for graph constructs absent from the source text.Absolute scoring gave RAG higher factual accuracy (0.424 vs. 0.231) and evidence grounding (0.432 vs. 0.200), while GraphRAG led clinical relevance (0.359 vs. 0.193).Source: C. Quantitative Evaluations · p. 4
- 76.4% of 5,864 criterion-level comparisons preferred GraphRAG, versus 4.1% for RAG and 19.5% ties.The paired comparative evaluation aggregates results across criteria.Source: C. Quantitative Evaluations · p. 5, Table II · p. 5
- GraphRAG won 93.9% of clinical-relevance comparisons, 84.0% of completeness comparisons, and 79.0% of evidence-grounding comparisons.Factual accuracy was its smallest comparative advantage: 48.8% GraphRAG wins, 46.4% ties, and 4.8% RAG wins.Source: C. Quantitative Evaluations · p. 5, Table III · p. 5
- GraphRAG win rates exceeded 80% on seven tasks, including risk factors (91.5%), ontology (89.4%), and comorbidity (85.1%).These tasks require multi-hop reasoning across heterogeneous node types and theoretical constructs.Source: C. Quantitative Evaluations · p. 5
- RAG remained competitive when evidence was self-contained in a few EHR fields; it led all four absolute criteria on suicide indicators, while ontology mapping and risk-balance tasks challenged both systems.GraphRAG had higher clinical relevance in 12 of 15 tasks in absolute scoring.Source: C. Quantitative Evaluations · p. 5
D. Discussion
Under a comparative protocol that checks each system against its own retrieved evidence, GraphRAG is preferred on most criterion-level comparisons, especially for clinical relevance and completeness. Its advantages are concentrated in relational reasoning, while suicide indicators and several high-level synthesis tasks remain challenging.
D. Discussion
- GraphRAG achieves a 79.0% evidence-grounding win rate, while factual accuracy is essentially tied at 48.8% under the corrected comparative protocol.The passage notes that flat-text ground truth can underestimate graph-derived constructs absent from source text.Source: D. Discussion · p. 5
- GraphRAG wins 76.4% of 5,864 criterion evaluations, with gains in clinical relevance (93.9%) and completeness (84.0%).The gains cluster in relational reasoning, including risk-factor identification, comorbidity analysis, risk-construct inference, protective-versus-risk balance, and summary synthesis.Source: D. Discussion · p. 5
- Suicide indicators favor RAG under both protocols, likely because dense clinical-note passages capture them more completely than distributed graph nodes.This pattern motivates a hybrid design pairing graph retrieval for relational inference with passage retrieval for dense clinical documentation.Source: D. Discussion · p. 5
- Both systems remain below 0.5 on protective-versus-risk balance, ontology mapping, and summary tasks, indicating unresolved high-level synthesis needs.The passage calls for stronger reasoning and grounding beyond retrieval architecture alone.Source: D. Discussion · p. 5
VII. CONCLUSION
BEACON-SP combines patient knowledge graphs and ontology-guided retrieval for structured reasoning over clinical, behavioral, social, and temporal evidence. On the benchmark, GraphRAG was preferred in 76.4% of head-to-head comparisons, and the paper identifies evaluation bias against graph-based systems.
VII. CONCLUSION
- BEACON-SP combines patient knowledge graphs with ontology-guided retrieval to reason over clinical, behavioral, social, and temporal evidence.Its ontology integrates three suicide-prevention theories into a unified representation of patient risk factors.Source: §VII · p. 5
- GraphRAG was preferred in 76.4% of head-to-head comparisons, with gains in clinical relevance (93.9%) and completeness (84.0%).The evaluation used 1,500 queries across 15 clinical categories and 100 patients.Source: §VII · p. 6
- The comparative protocol corrects bias against graph systems citing ontological constructs absent from flat EHR records.The paper presents BEACON-SP as a foundation for interpretable, evidence-grounded clinical decision support.Source: §VII · p. 6
- Future work will focus on larger benchmarks, clinician review, and hybrid graph-text retrieval.Source: §VII · p. 6