PaperlayerView exact revision on arXiv

Source-linked AI summary

From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

Bayram Ayadi, Florian Geissler, Johanna Schmidhuber

arXiv:2610.31467v1cs.CL

TL;DR

Wearable and digital health systems produce longitudinal physiological records, but translating them into accessible patient narratives risks semantic drift. This paper evaluates evidence-grounded, constrained generation with retrieved lay-term definitions for menstrual-health summaries. Across models, constraints improve uncertainty preservation, while readability worsens.

  • Problem

    Existing benchmarks focus on clinical text rather than patient-facing summaries generated directly from structured physiological observations, where faithfulness must be checked against raw evidence.

    Source: §2.1 · p. 2

  • Method

    The paper combines explicit evidence-grounding constraints and RAG with a taxonomy-based evaluation of structured-record summaries.

    Source: §1 · pp. 1–2, §3.5 · p. 4

  • Results

    Grounding constraints most consistently reduced epistemic misalignment across three models; retrieval improved faithfulness for GPT-4o and was neutral for two others, while constrained outputs were harder to read.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Patient-facing health summaries should be evaluated jointly for faithfulness, uncertainty preservation, coverage, and readability.

    Source: §6 · p. 8

    The five clinical-alignment metrics were not validated against a human-annotated subsample or clinician judgment.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As wearable and digital health systems increasingly generate longitudinal physiological records, communicating these observations in patient-accessible language remains a significant challenge. Large Language Models (LLMs) offer a promising mechanism for translating structured clinical information into readable narratives, but this process may introduce semantic drift, where clinically precise meaning is distorted during simplification. We investigate evidence-grounded health summarization for menstrual health monitoring using a constrained generation framework that combines structured evidence conditioning with Retrieval-Augmented Generation (RAG) using a prompt-based constrained generation framework. We evaluate semantic alignment using Natural Language Inference (NLI), atomic factuality metrics, and readability measures. Across three LLMs and 4,500 generated summaries, explicit grounding constraints consistently improve clinical alignment, particularly by improving uncertainty preservation and, once faithfulness is evaluated against retrieved evidence, generally preserving or improving faithfulness, while effects on omission remain model-dependent. We also find that improved clinical alignment comes at the cost of increased reading difficulty and medical terminology. These findings highlight the need to jointly evaluate faithfulness, uncertainty preservation, and readability when developing patient-facing medical LLM systems.

1 Introduction

Wearable health systems generate longitudinal physiological records that patients increasingly interpret themselves, but translating them into accessible language risks semantic drift. This paper develops evidence-grounded generation and evaluation for patient-facing menstrual-health summaries.

1 Introduction

  • Patients increasingly interpret longitudinal physiological data themselves, making accessible communication of observations an important challenge.These data span menstrual phases, HRV, skin temperature, and sleep quality.Source: §1 · p. 1
  • Existing simplification assumes a reference text, whereas patient narratives here must remain faithful to structured physiological observations and raw evidence.The gap is especially consequential in menstrual health, where cyclical signals and intrinsic uncertainty matter.Source: §1 · p. 1
  • The paper introduces a five-category taxonomy of semantic drift to measure misalignment as medical concepts move from clinical to public discourse.The taxonomy is part of a constrained generation framework intended to systematically measure and mitigate misalignment.Source: §1 · p. 1
  • The authors generate medically grounded menstrual-health data using a semi-synthetic framework.Source: §1 · p. 2
  • The proposed pipeline combines category-targeted evidence-grounding constraints with RAG-based retrieval of lay-term definitions, then evaluates narratives across the taxonomy.The evaluation is intended to derive preliminary insights into favorable configurations for menstrual-health monitoring.Source: §1 · p. 2

2 Related Work

Prior work studies simplification and clinical summarization, but existing benchmarks do not address patient-facing narratives generated directly from structured physiological observations. Related work also motivates claim-level verification while noting limitations of clinical NLI and entailment-based assessment.

2.1 Clinical Summarization and Lay-Language Simplification

  • Medical-to-lay simplification trades readability against fidelity, as clinically important qualifiers may be omitted or distorted.Reproductive-health language can further complicate simplification.Source: §2.1 · p. 2
  • Existing benchmarks simplify or summarize clinical text, not patient-facing narratives generated directly from physiological records and checked against raw evidence.Source: §2.1 · p. 2

2.2 NLI-Based Faithfulness Assessment and Hallucination Verification

  • Claim-level verification methods such as FActScore and AlignScore target subtle semantic drift better than surface metrics such as ROUGE.Atomic verification also depends on consistent fact decomposition.Source: §2.2 · p. 2
  • Clinical NLI has annotation artifacts and domain-shift limitations, and textual entailment is not equivalent to factual support.Prior work had not combined constrained generation, factuality verification, and longitudinal summarization for structured-to-narrative patient-facing wearable-health summaries.Source: §2.2 · pp. 2–3

3 Methodology

The paper formalizes patient-facing summaries of longitudinal physiological records as a task of preserving evidence, uncertainty, and accessibility. It evaluates constrained generation with retrieved lay definitions using a semi-synthetic menstrual-health dataset and multiple alignment and readability measures.

3.1 Task Definition

  • The task is to generate a lay-language summary that remains aligned with structured physiological evidence while preserving uncertainty and avoiding unsupported medical claims.The stated goal also includes making summaries understandable to non-expert users.Source: §3.1 · p. 3

3.2 Dataset Construction

  • The study uses 500 semi-synthetic single-day observations generated from a causal DAG, with profiles that simulate field availability and missingness.Confidence metadata reflects data completeness and phase clarity; the full pre-masking record is retained as ground truth.Source: §3.2 · p. 3

3.3 System Overview

  • RAG-Constrained generation adds retrieved lay-term definitions to explicit clinical communication constraints, while Base and Constrained settings omit retrieval.The retrieved resource is the README dataset of over 50,000 medical-term and lay-definition pairs.Source: §3.3 · p. 3

3.4 Experimental Setup and Baselines

  • Experiments sampled 500 records from the semi-synthetic menstrual health dataset, processing each across three models and three distinct generation configurations to produce 4,500 patient-facing summaries in total.The models were GPT-4o (OpenAI et al., 2024), Qwen2.5:14B-Instruct (Qwen et al., 2025), and Phi-4-mini-4B (Microsoft et al., 2025).Source: §3.4 · p. 4

3.5 Evaluation

  • Alignment evaluation measures entailment-based faithfulness, uncertainty-sensitive claims, unsupported diagnoses, omitted salient fields, and perceived urgency.The evaluation also measures readability, lexical and syntactic complexity, and medical jargon.Source: §3.5 · pp. 4–5

3.6 Statistical Procedure

  • Paired Wilcoxon tests compare settings within models and models within settings, with effect sizes guiding interpretation and Benjamini-Hochberg correction for multiple comparisons.Each test uses 500 matched pairs.Source: §3.6 · pp. 5–6

4 Results

Across the three models, explicit constraints most consistently reduce epistemic misalignment, while effects on faithfulness, omission, and urgency vary by model or setting. These alignment gains come with greater reading difficulty and more medical terminology.

4.1 Clinical alignment

  • Corrected diagnostic overreach is near zero in every cell after excluding pregnancy and ectopic pregnancy mentions, which manual review found were typically explanatory context.GPT-4o and Qwen2.5-14B are 0.000 in all settings; Phi-4-mini reaches 0.004 in base and 0.006 in rag_constrained.Source: §4.1 · p. 6
  • Omission drift improves for GPT-4o and modestly overall, but differs by model: Qwen2.5-14B rises slightly with retrieval and Phi-4-mini worsens monotonically.Pooled omission drift declines from 0.220 to 0.194 across settings.Source: §4.1 · p. 6
  • Urgency misalignment rises from base to constrained and then falls under rag_constrained for every model, returning GPT-4o and Phi-4-mini to base levels while Qwen2.5-14B only partially recovers.GPT-4o is consistently the least alarming model across settings, though its gap narrows with retrieval.Source: §4.1 · p. 6
  • Mepist (epistemic misalignment) decreases monotonically from base through constrained to rag_constrained across all three models, with pooled mean falling from 0.297 to 0.263 to 0.248.Median Mepist is 0.0 in every model-by-setting cell, while pooled average uncertainty-sensitive claims rise across settings from 1.17 to 1.76 to 2.70.Source: §4.1 · p. 6, Table 2 · p. 7
  • Mfaith is lowest or comparable under rag_constrained versus constrained for all models: GPT-4o improves from 0.186 to 0.152, Qwen2.5-14B shifts from 0.165 to 0.162, and Phi-4-mini rises from 0.212 to 0.220.All three models remain below their base levels; pooled Mfaith is 0.242 for base, 0.188 for constrained, and 0.178 for rag_constrained.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Readability and stylistic trade-offs

  • Grade-level readability worsens across settings for all three models, with the largest GPT-4o shift from base to constrained; further change under retrieval is modest.Dependency-tree depth follows the same pattern, suggesting explicit constraints rather than retrieval are the main driver of increased reading difficulty.Source: §4.2 · p. 7
  • Lexical-diversity effects depend on the model, while raw medical-entity counts rise under generation controls even as jargon density declines with longer outputs.GPT-4o shows a larger MTLD decline from base to rag_constrained than from base to constrained.Source: §4.2 · p. 7

5 Discussion

Grounding improves some dimensions of clinical alignment, most consistently epistemic alignment, but retrieval’s effects vary by metric and model. These gains come with increased reading difficulty and medical terminology.

5.1 Readability

  • Constraint-based outputs are harder to read; RAG-Constrained raises FKGL by d=0.26 versus Constrained, alongside near-doubled medical-term counts despite stable sentence length.FKGL, ARI, and SMOG measure sentence length and syllables rather than conceptual word difficulty; medical-term density per word declines even as absolute mentions increase.Source: §5.1 · pp. 7–8

5.2 Clinical Alignment

  • Across four alignment metrics, only epistemic misalignment improves consistently across all three models; retrieval’s other effects vary by metric.The authors attribute apparent retrieval-related diagnostic-overreach increases to vocabulary and report mixed, model-dependent omission effects.Source: §5.2 · p. 8
  • Correcting faithfulness to count retrieved context as evidence lowers RAG-Constrained Mfaith versus its original formulation for all three models (d = 0.76–0.80, all p < 10−40).After correction, RAG-Constrained is lower than Constrained for GPT-4o (d = 0.284, p = 1.07 × 10−10) and statistically indistinguishable for Phi-4-mini and Qwen2.5-14B (p > 0.15).Source: §5.2 · p. 8
  • Removing pregnancy and ectopic-pregnancy mentions reduces diagnostic overreach to ≤0.6% across every model and condition.Manual inspection attributed nearly all flagged cases to explanatory reproductive-context mentions rather than unsupported diagnoses.Source: §5.2 · p. 8
  • Omission effects are model-dependent: the full pipeline benefits GPT-4o, constraints alone benefit Qwen2.5-14B, and additional grounding worsens Phi-4-mini’s score.The omission metric checks direct statements of salient fields, so explanatory prose may fail its entailment check even when information is not truly lost.Source: §5.2 · p. 8

6 Conclusion

Constrained generation most consistently reduces epistemic misalignment, while retrieval’s effects on faithfulness, omission, and urgency differ. The authors call for joint evaluation and caution that the study’s synthetic data and automatic metrics do not establish clinical readiness.

6 Conclusion

  • Across three models, constrained generation most consistently reduces epistemic misalignment; retrieval improves faithfulness for GPT-4o and is neutral for the other two, while omission and urgency effects are smaller and model-dependent.Constrained and RAG-Constrained outputs also become harder to read, often less lexically diverse, and heavier in medical terminology.Source: §6 · p. 8
  • The paper recommends evaluating health summarization jointly for faithfulness, uncertainty preservation, coverage, and readability.It contributes a structured-to-narrative evaluation setting and semantic alignment framework for patient-readable explanations of physiological data.Source: §6 · p. 8

Limitations

  • None of the five automatic clinical-alignment metrics was validated against a human-annotated sample or clinician judgment.The authors identify human validation as the most important follow-up direction.Source: Limitations · p. 9
  • The semi-synthetic records and ground truth lack clinician review, and real-world self-tracked data may differ in noise, idiosyncrasy, and structure.The reported results characterize this input distribution rather than real user data; the authors distinguish biological validity from semantic consistency against each record.Source: Limitations · p. 9
  • Retrieval supplies lay-term definitions, not additional clinical evidence, so its measured contribution may not generalize to RAG systems retrieving clinical context.Source: Limitations · p. 9
  • Without an external baseline, the study cannot situate the pipeline’s absolute performance relative to other possible approaches.Source: Limitations · p. 9
  • Experiments cover menstrual health with semi-synthetic data, not other physiological domains or the full diversity of real-world presentations and demographics.The authors caution that findings may not transfer directly to other domains or clinically atypical records.Source: Limitations · p. 9

Ethical Considerations

  • The metrics are not clinically validated safety measures, and none of the tested configurations is presented as deployment-ready; released code supports further research and clinician review.Source: Ethical Considerations · p. 10

A Synthetic Data Generation: Causal DAG Specification

The synthetic menstrual-health dataset is generated by sampling variables in causal order, with cycle and luteal-phase lengths reflecting distinct variability assumptions. This structure is intended to make inter-variable correlations arise from shared physiological mechanisms rather than arbitrary co-occurrence.

A Synthetic Data Generation: Causal DAG Specification

A.2 Node 1: Cycle Length

  • Cycle length follows a right-skewed lognormal distribution, yielding a median of about 28.7 days, a mean of 29.3 days, and a standard deviation of 4.4 days.The distribution reflects population data documented in Bull et al. (2019), with more than 600,000 cycles.Source: §A.2 · p. 12

A.3 Node 2: Luteal Phase Length

  • Luteal-phase length is sampled from a tight normal distribution to represent its approximate constancy across cycles.This encodes the finding that inter-cycle variability is attributable almost entirely to the follicular phase.Source: §A.3 · p. 12

A.4 Node 3: Phase Boundary Derivation

The appendix specifies how the synthetic records are generated, masked, and checked, then documents the prompts and retrieval component used for summarization. It encodes linked physiological patterns and constrains retrieved definitions to support explanations without extending the structured record.

A.4 Node 3: Phase Boundary Derivation

  • Phase boundaries are derived deterministically, and a cycle day is then sampled uniformly from days 1 through the cycle length.Ovulatory and luteal boundaries are computed from the ovulation day and cycle length rather than assigned fixed day ranges.Source: §A.4 · p. 13

A.5 Node 4–5: Sleep (Phase-Conditioned)

  • Sleep quality is conditioned on the current phase, and sleep duration is subsequently sampled conditional on quality.The quality probabilities are presented as a conditional probability table; duration uses a truncated normal distribution.Source: §A.5 · p. 13, Table 3 · p. 13

A.6 Nodes 6–7: Autonomic Biometrics (Phase + Sleep Override)

  • Resting heart rate uses a phase-specific baseline plus a 4–8 bpm penalty for poor sleep, while poor sleep forces HRV trends into sympathetic-dominant states.The design encodes chronic luteal-phase elevation and acute sleep-deprivation effects on sympathovagal balance.Source: §A.6 · p. 13

A.7 Node 8: SpO2 (Independent)

  • SpO2 is sampled independently, while temperature trend follows phase-specific probabilities reflecting a postovulatory thermal shift.The temperature shift is attributed to progesterone thermogenesis.Source: §A.7 · p. 13, §A.8 · p. 13

A.9 Node 10: Symptoms (Causal Placeholder)

  • Symptoms are generated as consequences of upstream states, including sleep and autonomic conditions, or are gated by menstrual or luteal phase.Examples include fatigue triggered by poor sleep or elevated resting heart rate, and cramps gated to menstruation.Source: §A.9 · p. 13

A.10 User Profile Masking

  • Five user archetypes determine which fields are always present, probabilistically available, or never observed; confidence scores use diagnostic-field coverage and phase clarity.The clarity values vary by phase, with menstrual phase most identifiable and follicular phase least identifiable.Source: §A.10 · p. 14, §A.11 · p. 14

A.12 Causal Coherence Validation

  • With poor sleep, 100% of HRV trends are decreasing or erratic, while mean resting heart rate is 6–8 bpm higher than with good sleep.These checks are reported as validation of the hard HRV override and encoded sympathetic penalty.Source: §A.12 · p. 14

B System Prompts and Hyperparameters; C Retrieval-Augmented Generation Details

  • The prompts progress from a base patient-facing summary to constrained evidence and uncertainty rules, while RAG supplies lay definitions only as explanatory support for the structured record.The retrieval component indexes cleaned, deduplicated term-definition pairs; a generated example record is shown before masking, and the runs use temperature 0.4.Source: §B.1 · p. 14, §B.2 · p. 14, B System Prompts and Hyperparameters · p. 14

D Readability metrics

The section identifies the scope of the readability table, without providing its scores in the supplied passage.

D Readability metrics

  • Table 6 reports mean grade-level readability scores by model and generation setting.Source: Table 6 · p. 16

Appendix E: Medical Jargon-Related Metrics

Figure 2 presents distributions of medical-jargon measures across generation conditions and evaluated models.

Appendix E: Medical Jargon-Related Metrics

  • Figure 2 covers jargon density, raw medical entity counts, and entities per sentence across base, constrained, and rag_constrained settings for all evaluated models.Source: Figure 2 · p. 17, Table 7 · p. 17

F Metric correlations

Across 4,500 generated summaries, grade-level readability indices were strongly intercorrelated.

F Metric correlations

  • Grade-level indices were strongly intercorrelated (r ≥0.87) across 4,500 generated summaries.Figure 3 reports pooled pairwise correlations across seven readability, lexical, and syntactic-complexity metrics.Source: Figure 3 · p. 18

Related papers

This paper cites

Loading 2610.31467v1…