PaperlayerView exact revision on arXiv

Source-linked AI summary

MA-RAG: Multi-Agent Retrieval-Augmented Generation for Query-Driven Summarization of Longitudinal Parkinson's Disease Assessments

Sana Alamgeera, Denise Goberta, Muhammad Irshad, Anne H. H. Ngu

arXiv:2608.28624v1cs.CLcs.AI

TL;DR

Clinicians face challenges synthesizing structured Parkinson’s assessments across domains and visits. MA-RAG combines domain-specialized retrieval-augmented agents, structured fact extraction, and verification; it reports up to a 122% relative improvement in Fact Precision (0.436 to 0.990) and a 98% reduction in Hallucination rates (0.564 to 0.010) versus Traditional.

  • Problem

    Clinicians face substantial challenges synthesizing structured assessment data into actionable longitudinal insights; domains remain isolated, and visit-to-visit changes require manual comparison.

    Source: §1 · pp. 1–2

  • Method

    MA-RAG retrieves assessment facts into structured, domain-specific subsets, routes them to specialized agents, and verifies generated summaries against structured scores.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Against Traditional, MA-RAG achieved up to a 122% relative improvement in Fact Precision (0.436 to 0.990) and reduced Hallucination rates by up to 98% (0.564 to 0.010).

    Source: §6 · p. 13

  • Takeaways & Limitations

    MA-RAG produces clinically coherent summaries with strong factual grounding for Parkinson’s UPDRS and PDQ-8 assessments.

    Source: §6 · p. 13

    The current implementation is tailored to Parkinson’s disease assessments using UPDRS and PDQ-8, and evaluation on other clinical domains remains future work.

    Source: §5.3 · p. 13

Abstract

from arXiv · show

Accurate interpretation of single-visit and longitudinal clinical assessments for Parkinson's disease is time-consuming and often depends on specialist expertise. Although large language models (LLMs) can generate natural language summaries, they frequently lack domain-specific clinical grounding and struggle to produce factually correct and temporally consistent responses for structured longitudinal assessment data. To address these limitations, we propose MA-RAG, a query-driven multi-agent retrieval-augmented generation framework that decomposes clinical reasoning into domain-specialized agents, combines structured fact extraction, and synthesizes clinically grounded summaries through a final verification stage. The framework supports four clinical analysis tasks: single-session, trajectory, comparison, and cohort summarization. We evaluate MA-RAG using objective metrics, namely Fact Precision, Hallucination Rate, Temporal Fidelity, and Semantic Similarity, together with subjective evaluations conducted by clinical experts. Compared to Traditional, RAG-only, and Single-agent RAG baselines, MA-RAG substantially improves factual correctness, achieving up to a 122% relative increase in Fact Precision (from 0.436 to 0.990) and reducing the Hallucination Rate by up to 98% (from 0.564 to 0.010), while consistently receiving top ratings from clinical experts for organization and clinical usefulness. These results demonstrate that domain-specialized multi-agent reasoning enables reliable query-driven summarization of structured longitudinal clinical assessment data.

1. Introduction

Multidomain Parkinson’s assessments provide structured information, but manual scoring and cross-visit synthesis challenge clinicians. MA-RAG addresses these limits with metadata-based retrieval, domain-specialized agents, and verification against structured facts.

Assessment context

  • UPDRS assesses non-motor experiences, daily-living activities, and motor function, while PDQ-8 [2] complements these domains with patient-reported quality-of-life measures.UPDRS items use ordinal scores from 0 to 4, with higher values indicating greater symptom burden.Source: §1 · p. 1

Clinical challenge

  • Manual scoring, disconnected assessment domains, and cross-visit comparisons burden clinicians and make it difficult to form a unified view of changing Parkinson’s disease status.Clinicians may rely on static staging scales that do not fully reflect short-term changes.Source: §1 · pp. 1–2

Motivation

  • Standard LLMs may hallucinate numeric scores or neglect non-motor and quality-of-life findings, while RAG without structured reasoning can misorder longitudinal changes and lacks explicit fact-table verification.The introduction also warns that autonomous LLM or agentic reasoning without rule-based verification can amplify errors in high-stakes workflows.Source: §1 · p. 2

Approach

  • MA-RAG combines exact metadata retrieval, domain-specialized agents restricted to precomputed facts, and a final agent that verifies narratives against a structured score table before synthesis.The framework is designed for query-driven longitudinal summarization of multimodal Parkinson’s assessments.Source: §1 · p. 2

Evaluation

  • The framework is evaluated using Fact Precision, Hallucination Rate, Temporal Fidelity, and Semantic Similarity, alongside clinical-expert evaluations.Source: §1 · p. 2

2. Related Work

Clinical summarization progressed from rigid templates to fluent generators, retrieval, and agentic tools, but the related work identifies persistent grounding, temporal-alignment, verification, and evaluation gaps in longitudinal assessment summarization.

Related Work

  • Template-based NLG ensures factual consistency by construction but rigid patterns cannot synthesize cross-domain relationships or adapt to variable visit counts.Such templates can obscure clinically important discordance between domains.Source: §2 · p. 3
  • Transformer models, including ClinicalBERT and BioGPT, improve flexibility over templates but lack explicit grounding and achieve only 43.6% fact precision on longitudinal queries.They can invent values and change directions of progression by conflating similar score patterns across visits.Source: §2 · p. 3
  • RAG grounds single-session summaries in retrieved evidence but cannot ensure patient-visit alignment and leaves arithmetic aggregation and delta computation to the generator.Embedding similarity can retrieve records from other patients, undermining patient-specific temporal coherence.Source: §2 · p. 3
  • Agentic systems, including MedAgent and ClinicalAgents, use tools for score aggregation but lack explicit fact verification and retain embedding-based patient-mixing risks.A narrative can contradict correct tool output, and semantically similar records from different patients can still be retrieved.Source: §2 · p. 3
  • No prior framework unifies metadata-based retrieval, specialized agents, and fact-table verification for longitudinal multimodal summaries, while ROUGE, BLEU, BERTScore, and SBERT similarity miss factual and temporal errors.These metrics can score summaries highly despite errors in visit ordering, baseline anchoring, or numeric findings.Source: §2 · p. 3

3. Methodology … 3.3. Retrieve Node

MA-RAG turns natural-language queries over longitudinal patient records into clinically grounded summaries by planning the analysis, retrieving structured assessment facts, and routing domain-specific evidence for synthesis. Its methodology supports single-visit, longitudinal, cross-patient, and cohort questions while preserving requested temporal context and factual correctness.

3. Methodology

  • The framework combines a patient-record Knowledge Base and metadata-based Retrieve Node with query-interpreting LLM agents and a Final Agent that grounds summary synthesis in retrieved evidence.Its document-oriented representation is designed to support future integration of clinician-authored narratives through the same vector database and retrieval pipeline.Source: §3 · p. 4

3.1. Task Definition

  • MA-RAG reasons over domain-specific scores and item-level responses to produce summaries that preserve requested temporal context, integrate assessment domains, and maintain factual correctness.The assessments include instruments such as UPDRS and PDQ-8.Source: §3.1 · p. 4
  • The four supported tasks are single-session summaries, single-patient trajectories, cross-patient comparisons, and cohort analyses of score extremes, progression patterns, and risk profiles.Examples include summarizing a baseline visit, tracking UPDRS over visits, comparing patients, and identifying the highest UPDRS score.Source: §3.1 · p. 4

3.2. Orchestrator

  • The Orchestrator converts the natural-language query into a machine-readable plan specifying patient identifiers, analysis type, instruments of interest, and visit filter.Patient identifiers default to all available patients when the query refers to everyone or none can be recovered; the analysis type is selected using query keywords.Source: §3.2 · p. 5
  • If the plan cannot be parsed, the Orchestrator defaults to trajectory analysis, both instruments, and no visit filter, while the participant set constrains all downstream retrieval and reasoning.The resulting plan is passed to the next stage.Source: §3.2 · p. 5

3.3. Retrieve Node

  • The Retrieve Node maps the planned analysis type to an internal intent label using a fixed lookup table that reconciles minor naming differences.It receives the original query and plan and gathers the patient data needed to answer the query.Source: §3.3 · p. 5
  • For each planned patient, retrieval uses exact metadata matches for session records, extracts relevant domain and item-level scores, and organizes them by patient, visit, and domain.The fact table is divided into motor, ADL, nonmotor, and quality-of-life subsets so each downstream domain agent receives specialty-relevant facts.Source: §3.3 · p. 5
  • The Retrieve Node passes the intent, fact table, and domain-specific subsets to the Router; in the example, it extracts patient 1’s visit-1 Motor score of 14.The example uses a single-session baseline query for patient 1.Source: §3.3 · p. 5

3.4. Router … 3.7. Cohort Agent

Sections 3.4–3.7 describe how query intent routes data to specialized agents, which generate domain, patient-comparison, or cohort summaries for the Final Agent. Each path uses structured facts and domain reference material to support its narrative.

3.4. Router

  • The Router uses a rule-based function, without an LLM, to select the next stage from query intent: comparison, cohort/risk, or trajectory/single-session.Comparison intent selects the Comparison Agent; cohort or risk intent selects the Cohort Agent; trajectory or single-session intent selects the Domain Agents.Source: §3.4 · pp. 5–6
  • Comparison intent sends the full fact table to the Comparison Agent, while trajectory and single-session intents send each Domain Agent only its corresponding domain facts.The routed output identifies the invoked agent and the data passed to it.Source: §3.4 · p. 6

3.5. Domain Agents

  • Four Domain Agents run sequentially on their own fact subsets, the query, and static scale references, skipping any domain with no facts.Each agent uses an LLM to write a domain progress narrative.Source: §3.5 · p. 6
  • Domain Agents calculate visit-to-visit and baseline score changes, label decreases as improvement and increases as worsening, then pass domain progression summaries to the Final Agent.They compute changes for each visit after the first.Source: §3.5 · p. 6

3.6. Comparison Agent

  • The Comparison Agent builds a rule-based table of every patient's domain scores at each visit and marks each score as improved, worsened, or unchanged from the prior visit.It combines this table with the query and all four domain scale references in an LLM-generated cross-patient summary passed to the Final Agent.Source: §3.6 · p. 6

3.7. Cohort Agent

  • For population-level questions, the Cohort Agent computes each patient's domain progression slope and cohort-wide maximum, minimum, and average scores, then uses query keywords and fixed rules to select a statistic of interest.Examples include mapping “fastest” plus a domain to its steepest slope and “stable” to the smallest absolute slope.Source: §3.7 · p. 6
  • The Cohort Agent combines the selected statistic, query, and all four domain scale references in an LLM-generated clinical explanation, which it passes to the Final Agent.Source: §3.7 · p. 6

3.8. Final Agent

The Final Agent verifies the upstream narrative against a fact table before producing a clinical summary, with synthesis tailored to cohort, comparison, single-session, or trajectory mode.

3.8. Final Agent

  • The agent builds a score table from the fact table and flags visits with a zero score in one domain and an elevated score in another for closer verification.It checks the upstream narrative against the score table and underlying facts so that only confirmed facts are included in the final summary.Source: §3.8 · p. 7
  • A dispatcher selects cohort, comparison, single-session, or trajectory synthesis mode based on which upstream agent produced output and the planned analysis type.The Final Agent uses the matching mode-specific LLM call to verify the corresponding cohort, comparison, or domain-agent narrative.Source: §3.8 · p. 7
  • Trajectory mode summarizes baseline-anchored progression using cumulative change, then cross-domain patterns and monitoring needs; comparison mode contrasts patients by domain and identifies discordance and divergence.The final clinical summary is returned to the user in response to the query.Source: §3.8 · p. 7

4. Implementation Details

MA-RAG uses longitudinal UPDRS and PDQ-8 assessments, organizes visit-level records into clinical domains, and combines a structured retrieval knowledge base with agent-specific clinical context. Its implementation also specifies the model and workflow setup and an offline evaluation protocol based on clinician-assisted gold summaries and four metrics.

4.1. Datasets

  • The dataset contains 44 adults with Parkinson’s disease assessed at four sessions over 24 weeks using clinician-rated UPDRS and patient-reported PDQ-8 measures.Participants were diagnosed at Hoehn and Yahr stages 1–3 at baseline; mean age was 67.1 years and mean disease duration was 4.3 years. Records are chronologically indexed by visit and organized into motor, ADL, non-motor, and QoL domains.Source: §4.1 · p. 7

4.2. Knowledge Base Construction

  • Visit documents encode patient, visit date and index, domain scores, and item scores, then are embedded and stored with metadata in ChromaDB for exact-match retrieval.Metadata identifies the instrument and whether a document represents a session or longitudinal summary; single-visit retrieval can filter patient records to the baseline visit.Source: §4.2 · pp. 7–8

4.3. Static Clinical Context

  • Each domain-specific agent receives static clinical context covering assessment-item definitions, scoring guidance, interpretation, and reliability statistics drawn from UPDRS and PDQ-39.This context is not retrieved from the knowledge base because it is invariant across queries and retrieval may truncate clinically important information.Source: §4.3 · p. 8

4.4. Experimental Setup

  • The system uses ChromaDB, all-MiniLM-L6-v2 embeddings, Llama-3.1-8B-Instruct, greedy decoding, and a LangGraph workflow, served through Gradio on one NVIDIA A100 GPU.Generation is capped at 1,200 tokens with a 1.1 repetition penalty.Source: §4.4 · p. 8

4.5. Evaluation Protocol

  • Offline evaluation compares summaries with clinician-assisted GPT-5.5 gold summaries using Fact precision, hallucination rate, temporal fidelity, and SBERT similarity.Fact precision measures numeric tokens also present in an extracted fact table; hallucination rate measures generated numeric tokens absent from that table. Temporal fidelity checks visit-index order, and SBERT similarity is cosine similarity between generated and gold summary embeddings.Source: §4.5 · pp. 8–9
  • The four metrics are computed per query and reported as means grouped by single-session and longitudinal task type.Source: §4.5 · p. 9

5. Results and Discussion

Across offline evaluations, MA-RAG achieved the strongest factual performance, while clinicians rated its summaries positively and identified longitudinal summarization as the main area for improvement. The framework’s added inference cost varies by analysis type, and its current clinical implementation is tailored to Parkinson’s disease assessments.

5.1.1. Comparison Across Different Frameworks

  • For longitudinal queries, Fact Precision rose from 0.436 (Traditional) to 0.652 (RAG-only), 0.780 (Single-agent RAG), and 0.970 (MA-RAG) on UPDRS; on UPDRS+PDQ-8, it rose from 0.592 to 0.652, 0.765, and 0.990, respectively.MA-RAG also achieved the highest Fact Precision and lowest Hallucination Rate across single-visit and longitudinal tasks; the authors attribute the gains to retrieval, structured fact extraction, domain-specialized agents, and final verification.Source: §5.1.1 · p. 10, Table 1 · p. 10, Table 2 · p. 10
  • All paired t-test comparisons of MA-RAG against each baseline across tasks and metrics were significant (p < 0.05), while Temporal Fidelity and SBERT Similarity alone did not reliably indicate factual correctness.For single-visit UPDRS+PDQ-8, Single-agent RAG had the highest Temporal Fidelity (0.700) despite a 0.180 hallucination rate, and Traditional RAG had the highest SBERT similarity (0.634) despite Fact Precision of 0.639.Source: §5.1.1 · p. 10

5.1.2. Comparison Across Different LLMs

  • Across evaluation tasks, Llama-3.1-8B-Instruct achieved the highest Fact Precision and lowest Hallucination Rate, with longitudinal Fact Precision of 0.970 (UPDRS) and 0.990 (UPDRS+PDQ-8) and hallucination rates of 0.020 and 0.010.It also maintained single-visit Fact Precision scores of 0.940 and 0.930, outperforming the specialized biomedical models on factual performance.Source: §5.1.2 · p. 11, Table 3 · p. 11, Table 4 · p. 11
  • Perfect or near-perfect Temporal Fidelity did not ensure factual accuracy: Meditron-7B reached up to 1.000 but had longitudinal UPDRS Fact Precision of 0.513, whereas Llama-3.1-8B-Instruct better adhered to structured facts.The authors note that generalist instruction-following capabilities helped Llama-3.1-8B-Instruct follow the structured fact table across specialized-agent prompts.Source: §5.1.2 · p. 11

5.2. Subjective Evaluation

  • Clinicians rated summaries across all four query categories positively, with mean scores of 3.67–5.00; Cohort summaries scored 4.67–5.00, while Trajectory summaries scored 3.67–4.17 and were weakest in clinical actionability.Single Session organization received 4.83; three clinicians evaluated eight summaries using six five-point Likert questions.Source: §5.2 · pp. 11–12, Figure 3 · p. 12
  • Clinicians praised factual content and organization, especially for Cohort and Comparison summaries, while recommending treatment context, stronger cross-domain reasoning, and clearer, more concise language for Trajectory summaries.These suggestions target the longitudinal read-interpret-decision workflow.Source: §5.2 · p. 12

5.3. Computational Cost, Ethical Considerations and Limitations

  • Trajectory and Single-Session analyses require six LLM calls (6.0× single-pass RAG cost), whereas Comparison and Cohort require three calls (3.0×); MA-RAG runs an open-source LLM locally and restricts input to predefined query templates.The authors state that local deployment keeps sensitive clinical data within the institution and that the added computation is allocated to more demanding tasks.Source: §5.3 · pp. 12–13, Table 5 · p. 13
  • The current implementation targets Parkinson’s disease assessments using UPDRS and PDQ-8, but its modular architecture can be adapted to other diseases by changing the domain-specific reasoning layer.Future work will evaluate MA-RAG on multimodal assessment datasets from additional clinical domains.Source: §5.3 · p. 13

6. Conclusion ad Future Work

MA-RAG combines domain-specialized agents, structured fact extraction, and final verification to generate factually grounded Parkinson’s assessment summaries. The conclusion reports improved benchmark performance and clinical-expert ratings, and outlines planned extensions and validation.

Conclusion

  • Fact Precision improved by up to 122% (0.436 to 0.990) and Hallucination rates fell by up to 98% (0.564 to 0.010) versus the Traditional baseline.Offline evaluations also found MA-RAG consistently outperformed Traditional, RAG-only, and Single-agent RAG baselines.Source: §6 · p. 13
  • Clinical experts rated MA-RAG summaries clinically meaningful and well organized across all four analysis types.The conclusion presents these evaluations as supporting the framework’s potential as a clinical decision-support tool for longitudinal patient assessment.Source: §6 · p. 13

Future Work

  • Future work will extend MA-RAG beyond Parkinson’s disease to other neurological disorders and heterogeneous clinical data sources.Planned sources include electronic health records, imaging reports, laboratory results, and wearable sensor data.Source: §6 · p. 13
  • The authors plan memory-augmented adaptation using clinician feedback to improve longitudinal reasoning without fine-tuning the underlying LLM.Source: §6 · p. 13
  • The framework will be validated in larger prospective clinical studies involving more clinicians, institutions, and real-world deployment settings.Source: §6 · p. 13

A. Appendix · B. Static Clinical Context

MA-RAG supplies domain agents with static clinical contexts that encode domain-specific assessment knowledge. The combined context replaces Knowledge Base retrieval to provide complete, consistent domain knowledge at inference time.

B. Static Clinical Context

  • Static contexts cover motor, ADL, nonmotor, and QoL domains, encoding scale definitions, item-level scoring rules, clinical factor domains, and reliability statistics for each domain agent.Figure 4 presents these domain contexts.Source: B. Static Clinical Context · p. 16, Figure 4 · p. 16
  • The combined static context is distributed to corresponding agents in place of Knowledge Base retrieval.This supplies the domain knowledge directly to each agent at inference time.Source: B. Static Clinical Context · p. 16
  • This arrangement ensures complete and consistent domain knowledge at inference time.Source: B. Static Clinical Context · p. 16

C. System Prompts

The orchestrator prompt converts a user’s Parkinson disease question into a structured analysis plan. It specifies participant, analysis type, metrics, and visit filter fields with rules for selecting their values.

C. System Prompts

  • The plan identifies query-mentioned patient IDs or ALL, and specifies an analysis type, UPDRS and/or PDQ8 metrics, and a visit filter.Allowed analysis types are trajectory, comparison, cohort, and single-session; filters include none, baseline, latest_visit, baseline_vs_latest, and first_N_visits or last_N_visits with the requested N.Source: C. System Prompts · pp. 16–17
  • Keyword rules route comparison, trajectory, cohort, and single-session questions to their corresponding analysis types.Examples include “vs” for comparison, “over time” for trajectory, “highest” for cohort, and “baseline” or “most recent” for single-session.Source: C. System Prompts · p. 17
  • Filter rules use all visits for trajectory or comparison, and select baseline, latest, baseline-versus-latest, or the specified first or last N visits when requested.Metric rules select UPDRS for motor, ADL, or non-motor domains; PDQ8 for quality of life; and both for general queries.Source: C. System Prompts · p. 17
  • The prompt requires the system to return only a plan using the exact PARTICIPANTS, ANALYSIS_TYPE, METRICS, and FILTER format; rules limit analysis type to four categories and metrics to UPDRS or PDQ8.Source: C. System Prompts · pp. 16–17

D. Example Outputs

The examples show query-driven outputs for longitudinal trajectories, baseline status, cohort risk, and between-patient comparisons, combining domain scores with clinical interpretations and monitoring priorities.

Trajectory

  • For Patient 1’s trajectory, the summary flags Motor—particularly bradykinesia—and ADL for monitoring, noting incomplete bradykinesia resolution and severe Visit 4 impairment.It characterizes Non-Motor worsening as the earliest warning sign and ADL recovery as the slowest.Source: D. Example Outputs · p. 18

Single Session

  • At baseline, Patient 1 had Motor, ADL, Non-Motor, and PDQ-8 total scores of 14, 13, 13, and 6, respectively.The summary highlights sleep problems and daytime sleepiness for follow-up, alongside ADL difficulties involving speech, handwashing, and salivation.Source: D. Example Outputs · pp. 18–19

Cohort

  • The cohort summary identifies Project ID 20 as high risk due to the largest motor increase of 16 points and fastest QOL worsening slope of 8.0.These are the stated reasons for its high-risk designation.Source: D. Example Outputs · p. 19

Comparison

  • The comparison describes Patient 4’s ADL fluctuations and motor–ADL divergence, while Patient 7’s PDQ8 worsens consistently and warrants monitoring.It also notes Patient 4’s better PDQ8 score than Patient 7 at some visits despite worse motor symptoms, and identifies Visits 5–6 as potential inflection points.Source: D. Example Outputs · p. 19

Related papers

Loading 2608.28624v1…