PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Clinical Vignettes that Preserve Cognitive Formulations

Amit Oren, Nimrod Hertz-Palmor, Dean Ariel, Guy Laban

arXiv:2608.29995v1cs.CLcs.AI

TL;DR

Clinical vignette generation needs fidelity to person-specific clinical structure, not fluency alone. FORMA uses cognitive formulations as graph-based generation specifications and validates their realization in PTSD vignettes. Full-condition vignettes preserved specified structure better than zero-shot alternatives across structural, expert, and clinician evaluations.

  • Problem

    LLMs can generate fluent clinical vignettes but may average over patients and reproduce trauma-type templates instead of preserving specifiable, person-specific clinical structure.

  • Method

    FORMA compiles a disorder’s cognitive model into a directed weighted graph, samples graph-based personas, conditions generation on them, and validates components and causal links.

  • Results

    Full-condition edge recovery reached MCC = +0.41 and AUC = 0.70, whereas zero-shot recovery was MCC = +0.01 and AUC = 0.50; experts and clinicians also favored full vignettes.

  • Takeaways & Limitations

    Cognitive formulation can serve as an auditable specification for scalable synthetic clinical text generation.

  • Takeaways & Limitations

    The fairness analysis tests output equivariance but not stereotyping in content, representation harms, or intersectional disparities, and clinician-study generalizability remains uncertain.

Abstract

from arXiv · show

Large language models can generate fluent clinical case vignettes, but fluency alone does not ensure fidelity to a specifiable clinical structure. We introduce FORMA, a theory-grounded framework that compiles a cognitive model of a disorder into a directed weighted graph, samples a person-specific configuration of that graph, and validates whether the generated vignette preserves the specified components and causal links. We instantiate FORMA on Posttraumatic Stress Disorder using the Ehlers and Clark cognitive model, generating 16,500 vignettes across 500 personas, 11 generation models, and three ablation conditions. Evaluation combines an external edge-recovery probe, two clinical experts, a scaled LLM judge, and a clinician user study with 100 licensed practitioners. The cognitive graph is recoverable from full-condition vignettes (MCC = +0.41, AUC = 0.70) but not from zero-shot generation (MCC = +0.01, AUC = 0.50). Experts rate full vignettes substantially higher than zero-shot alternatives, and clinicians perceive them to be human-written 85% of the time, compared with 22% for zero-shot. FORMA also reduces demographic disparity in perceived quality by 1.5-7x. These results show that cognitive formulation can serve as an auditable specification for scalable synthetic clinical text generation. A repository with the data and code is available online: https://github.com/Amit-Oren/FORMA.

1 Introduction

FORMA addresses the gap between fluent clinical vignettes and fidelity to person-specific cognitive structure. It compiles cognitive formulations into graphs, generates structured personas and vignettes, and evaluates whether causal links survive generation.

  • Motivation: Clinical vignettes must resemble real clinical material while varying beyond the small, prototypical cases available in hand-curated banks.LLM generation offers scale, but diagnostic-label prompting can average over patients and reproduce trauma-type templates.
  • Motivation: Cognitive formulation represents disorders as person-specific directed networks whose interacting components explain maintenance, rather than as symptom checklists.The framework draws on formal cognitive models, including the Ehlers and Clark model of PTSD.
  • Contributions: FORMA compiles a cognitive model into a directed weighted graph, samples graph configurations as personas, and uses closed-loop validation to enforce structural fidelity.Its ablation design separates graph structure from self-report items and demographic specification.
  • Contributions: FORMA’s evaluation triangulates structural validity, expert ratings, scaled LLM judgments, and clinician perceptions across multiple generation models.The study also includes a three-condition comparison isolating the marginal contribution of formulation.
  • Resources: The released dataset contains 16,500 vignettes spanning 500 personas, 11 models, and three conditions.The evaluation subsample includes expert and LLM-judge ratings, with additional clinician user-study ratings.

2 Related Work

Prior work established the value of clinical vignettes and explored LLM-based generation, but it generally emphasized fluency or informal scaffolds rather than formal cognitive graphs and directed causal fidelity.

  • Prior vignette generation: Clinical vignettes support training, research, and assessment, while unvalidated vignettes can distort clinical reasoning.Recent LLM efforts produce fluent outputs but remain structurally shallow.
  • Research gap: Existing work does not use a formal cognitive model as the generation specification or evaluate preservation of directed causal structure.This identifies the specific gap addressed by FORMA.
  • Research gap: Case formulation supplies a person-specific directed network, and graph-to-text methods provide relevant generation machinery, but this combination had not been applied to clinical case generation.Recent mental-health NLP work uses diagrams, expert principles, or reframing operators without formalizing the scaffold as a graph.
  • Positioning: FORMA is positioned at the intersection of established case formulation and structured graph-to-text generation.The paper applies mechanistic cognitive models as the structured knowledge source for clinical text.

3 Methodology

FORMA constructs validated personas from sampled demographics, self-report items, and Ehlers–Clark cognitive graphs, then generates and repairs vignettes against specified causal links.

  • Persona construction: The PTSD model is operationalized as a directed graph over five components: trauma memory, re-experiencing triggers, current threat, negative appraisals, and maladaptive strategies.Each persona receives a sampled graph configuration and matched self-report items from a clinical item pool.
  • Persona construction: Demographics and cognitive structure are sampled independently, while active components receive three self-report items and undergo plausibility validation.Failed demographic or persona fields are resampled or repaired through validator-directed processing.
  • Vignette generation: The Crafter receives demographics, self-report items, active components, and weighted causal links to produce a 500–700-word third-person narrative.Edge weights determine whether relationships are expressed explicitly, implicitly, or only in passing.
  • Validation: The Validator checks active components and positive-weight edges, and the Crafter inserts targeted patches for unsatisfied edges for up to five retries.The loop preserves existing text by adding an anchored sentence and a new sentence rather than rewriting the vignette.
  • Ablations: Full, no-formulation, and zero-shot conditions isolate the contributions of the graph, self-report items, demographics, and validation.Only the full condition includes the cognitive graph and structural validation on active edges.

4 Experimental Setup

The experiment evaluates 500 clinically severe PTSD personas across diverse demographics, trauma categories, generation models, conditions, rating dimensions, and validation procedures.

  • Personas: 500 personas span ages 18–80, approximately balanced gender, eight world regions, 29 trauma categories, and clinically elevated PCL-5 severity.The realized demographic and clinical attributes were tested for mutual independence.
  • Generation design: Each persona is generated under three conditions using 11 models from five model families.The conditions are full, no-formulation, and zero-shot.
  • Rating schema: Vignettes are scored across 18 items covering perceived quality, narrative quality, DSM-5 PTSD criteria, and Ehlers–Clark components.The component items are the primary locus for testing formulation effects.
  • Validation: The validation loop targets unsatisfied edges and allows up to five Crafter retries before evaluation.The Validator identifies violations and passes them to the Crafter for targeted repair.
  • Evaluation procedure: Experts calibrate before blind scoring, while agreement and condition contrasts use binary and ordinal reliability and statistical tests.Ground-truth recovery compares expert component labels with each persona’s specified active nodes.

5 Results

FORMA preserves person-specific cognitive components and directed relationships in generated PTSD vignettes, while improving cross-model consistency, expert ratings, clinician-perceived quality, and fairness relative to ablated conditions.

  • Structural Fidelity: All five cognitive components recover above chance in full-condition vignettes but collapse to chance under zero-shot generation.Expert component recovery in full reached AC1 ranges of 0.64–0.86 for R1 and 0.66–0.85 for R2; scaled-judge recovery reached 0.51–0.75.
  • Structural Fidelity: MCC = +0.41 and AUC = 0.70 for full-condition edge recovery, compared with chance-level MCC = +0.03 without formulation and +0.01 in zero-shot.Specified-active edges were judged present 75% of the time versus 35% for specified-absent edges in full, a 40.8-percentage-point gap.
  • Structural Fidelity: All 20 directed cognitive edges have positive MCC in full, ranging from +0.25 to +0.66, with strongest recovery for model-emphasized maintaining relationships.Maladaptive Strategies →Threat reaches MCC +0.66; Triggers →Negative Appraisals is also among the strongest reported edges.
  • Human Validation: Human raters reproduce the formulation-dependent graph signal, with full-condition MCC from +0.32 to +0.41 and AUC from 0.65 to 0.69 versus chance in both ablations.Human-to-LLM agreement is at least as high as agreement between the two human raters themselves.
  • Vignettes’ Geometric Structure: Full-condition generations converge more strongly across models, with MPNet cosine similarity μ = 0.792 versus 0.737 in zero-shot and full-vs-zero-shot dz = 1.52.The ordering and contrast replicate with BGE, whose full-condition mean is 0.837 and full-vs-zero-shot dz = 1.61.
  • Vignettes’ Geometric Structure: The formulation shifts latent organization from trauma-type clustering toward persona-specific cognitive signatures, raising the cohesion ratio from 0.07 in zero-shot to 0.48 in full.This is approximately a 7× shift in the reported cognitive-signature/trauma-category cohesion ratio.
  • Clinical-Content Quality: Inter-rater agreement on cognitive-formulation items is strongest in full and collapses in zero-shot, while DSM-5 PTSD coverage remains 0.85–1.00 across conditions.Threat All-5 AC falls from 0.72 to 0.10 and Strategies from 0.88 to 0.11; the diagnostic floor is therefore maintained while structural agreement changes.
  • Clinical-Content Quality: Experts rate full vignettes higher on formulation-sensitive content, with Importance 2.94 versus 2.23 in zero-shot and dz = +1.98.The largest expert gaps occur for Threat, Appraisals, Memory, and Strategies, whereas DSM-5 items remain near ceiling.

6 Discussion and Conclusion

FORMA changes the structural content of generated clinical vignettes, not merely their surface style. Its cognitive graph supports recovery, model convergence, higher expert ratings, and more frequent human-like clinician judgments.

  • The specified directed graph is recoverable from full-condition vignettes but not from zero-shot or no-formulation vignettes.
  • FORMA makes different generation models converge on a shared portrayal of the same persona while breaking zero-shot trauma-type stereotyping.
  • Clinical experts rate full vignettes higher than zero-shot alternatives.
  • Clinicians perceive full-condition vignettes as human-written far more often than zero-shot vignettes.
  • A structured cognitive-formulation graph can serve as an external gold for generated text and make the formulation effect identifiable.

Limitations

The framework’s validation and generalisation claims are bounded by shared-model self-validation and synthetic, culturally narrow data and evaluation settings.

  • The Vignette Validator shares the Crafter’s LLM, so iteration counts reflect both generation ability and self-critique.Qwen rows passing on the first attempt illustrate the risk of fluent self-approval.
  • Demographics and cognitive formulation graphs are sampled programmatically rather than representing real patients.The framework is a controlled stress test, and its faithfulness to actual clinical heterogeneity remains unanswered by this dataset.
  • The framework’s behaviour under translation or culturally specific persona sampling is untested.
  • The fairness analysis tests output equivariance on recorded variables, not deeper representation harms, stereotyping, or intersectional disparities.The authors caution against using the dataset to evaluate clinical bias in downstream LLMs before further axes are audited.

7 Ethical Considerations

FORMA’s ethical case rests on synthetic data without real patient material, but its apparent human authorship and theoretical commitments create downstream risks requiring disclosure and restraint.

  • The dataset contains no real patient material; personas, demographics, self-report items, and vignettes are LLM-generated.
  • Full-condition vignettes passing as human-written 85% of the time creates a mis-attribution risk in training and evaluation materials.The authors recommend prominently and persistently labeling every vignette as synthetic and as not representing a real individual.
  • Downstream models trained on FORMA vignettes may inherit the framework’s theoretical commitments and generators’ stylistic regularities.
  • Faithfulness to one cognitive model can propagate its framings, blind spots, and underrepresented populations.The Ehlers and Clark model is described as validated for PTSD maintenance in Western clinical samples, not as a theory of culturally specific trauma response or complex PTSD.
  • The fairness analysis does not test stereotyping in textual content or deeper representation harms.
  • The dataset and code are released openly, with an interactive browser showing each vignette alongside its persona graph and self-report items.

F Mean Token Counts by Model and Condition

Token counts reflect both graph specification and validation overhead and large differences in model verbosity, while one model’s extreme count reflects control-flow failure rather than quality.

  • The formulation graph adds roughly 2,000–4,000 tokens of specification content to each full-condition generation.The validation loop can append Crafter-generated patches until structural coverage is achieved.
  • Generation models differ widely in first-pass verbosity, producing an approximately 14× range in full-condition tokens.Gemini models exceed 2,700 tokens even in zero-shot, while Qwen models produce roughly one-third as many.
  • 29,234 tokens for gpt-4o-mini in the full condition reflect repeated refinement after invalid patch markers, not a structurally rich vignette.The model is excluded from per-condition analyses because the Validator repeatedly flags the same edge violations.
  • Token count is not a quality measure.The paper reports token counts for reproducibility, inference-cost budgeting, and transparency about graph-specification cost, not for filtering or weighting analyses.

I Expert-Annotated Vignette (Persona 14)

The expert-annotated Persona 14 vignette instantiates all five Ehlers–Clark PTSD components and their predicted maintenance cycle. Across raters, component recovery is positive in the full condition but collapses toward chance in zero-shot generation.

  • Expert-Annotated Vignette (Persona 14): The Persona 14 vignette represents a 54-year-old female combat engineer with all five Ehlers–Clark components active under the full condition.The persona includes Triggers, Negative Appraisals, Memory, Threat, and Maladaptive Strategies.
  • Expert-Annotated Vignette (Persona 14): The annotation identifies triggers, fragmented trauma memory, catastrophic appraisals, avoidance and numbing strategies, and bodily threat readiness.Examples include heat, flames, shouting, hypervigilance, and emotional blunting.
  • Expert-Annotated Vignette (Persona 14): The vignette expresses the predicted maintenance cycle from triggers through memory, appraisals, strategies, and renewed threat.The annotator marks the cycle as a concrete narrative pattern rather than merely a checklist of symptoms.
  • Per-Rater Recovery: Full-condition component recovery is positive for every rater, with every full-block MCC above zero and AC1 ranging from 0.60–0.92.Human experts recover components on par with or above LLM judges on most components.
  • Per-Rater Recovery: Zero-shot component recovery collapses across raters, with human-expert AC1 becoming negative for Memory, Threat, and Appraisals.Triggers retain slight positive LLM-rater agreement because they are frequently mentioned, a base-rate effect.
  • Caveat: The authors caution that the no-formulation block is mixed because experts may weigh explicit mentions and implicit context differently.LLM raters apply the explicit-mention criterion more strictly.

L Per-Edge Recovery in the Full Condition

Per-edge analysis tests whether the full-condition text preserves individual directed Ehlers–Clark links rather than merely co-mentioning components. All 20 edges recover above chance, with strongest recovery concentrated in the theory’s maintenance-cycle links, while probe precision is weaker for ambiguous directions.

  • Per-Edge Recovery: All 20 directed edges recover above chance in full-condition vignettes, with the strongest results on four maintenance-cycle edges.The strongest edges are Strategies →Threat, Triggers →Negative Appraisals, Threat →Memory, and Threat →Negative Appraisals.
  • Per-Edge Recovery: Strategies →Threat reaches MCC +0.66, Triggers →Negative Appraisals +0.63, Threat →Memory +0.57, and Threat →Negative Appraisals +0.56.These links are explicitly identified by the Ehlers–Clark theory as maintaining PTSD.
  • Per-Edge Recovery: Triggers →Threat has MCC +0.34 with recall 1.00 but precision 0.47, while Threat →Strategies has MCC +0.25 and precision 0.56.The probe often infers direction from co-occurrence of the two endpoints.
  • Caveat: The edge gold standard treats any positive weight as active, ignoring weight magnitude and creating a binary w > 0 versus w = 0 distinction.Within active edges, recovery probability is roughly flat across weight bands at 0.74–0.84.
  • Human Validation: Human and LLM raters show pairwise presence agreement between 0.63 and 0.75 on the 150-vignette edge-annotation sample.Human-to-LLM agreement is at least as high as agreement between the two human raters.
  • Human Validation: Human raters recover the specified graph in full with MCC +0.318 to +0.408, versus MCC −0.023 to +0.056 under both ablations.Full-condition AUC is 0.645 to 0.689, compared with 0.487 to 0.535 under the ablations.
  • Threshold Sensitivity: Raising the positive-edge threshold from w > 0 to w > 0.5 lowers full-condition MCC from +0.41 to +0.22 but leaves ablations at chance.The full-versus-zero-shot gap remains positive at every tested threshold.

N.2 Judge identity for the component-recovery analysis

Component-recovery findings are robust across judge identities, encoder choices, and matched-length analyses. Full-condition recovery remains positive across judges, while zero-shot recovery is generally at chance and formulation effects on key structural items persist after controlling for vignette length.

  • Judge Identity: Every full-condition component-recovery cell is positive across Opus, Flash, and DSpro and all five components.The recovery direction is therefore robust to the identity of the judge.
  • Judge Identity: Mean full-condition MCC is +0.44 for Opus, +0.35 for Flash, and +0.30 for DSpro.Opus is most sensitive and DSpro most conservative, while component rank-order remains consistent.
  • Judge Identity: Every zero-shot judge cell is at or near chance, with mean MCC +0.04 for Opus, −0.08 for Flash, and +0.06 for DSpro.The full-versus-zero-shot MCC gaps are +0.40, +0.43, and +0.24 respectively.
  • Encoder Sensitivity: BGE reproduces the same full > no-formulation > zero-shot convergence ordering as MPNet, with full-versus-zero-shot Cohen’s dz = +1.61 under BGE versus +1.52 under MPNet.The result is reported as evidence that convergence is not specific to MPNet’s training.
  • Length Sensitivity: Matched-length pairs retain large formulation effects for Maladaptive Strategies dz = +0.72, Negative Appraisals +0.53, and Narrative quality +0.57.The median word-count difference across 4,935 matched pairs is 16 words.
  • Length Sensitivity: With log(word_count) controlled, formulation coefficients retain 92.4% of their Strategies magnitude and 93.7% of their Appraisals magnitude.Narrative retains 83% and Memory 68%, while Threat shows substantial mediation through formulation-driven richer description.
  • Cluster Geometry: Zero-shot vignettes cluster ∼30% more tightly by trauma type, whereas full-condition vignettes cluster ∼5× more tightly by cognitive formulation signature.The composite ratio is 0.07 in zero-shot, 0.22 in no-formulation, and 0.48 in full.
  • Cluster Geometry: The BGE t-SNE replication shows tight zero-shot trauma-type clusters dispersing under Full and No-formulation across the representative models.This qualitative pattern replicates the body figure under a structurally distinct encoder.

O Trauma-Type Stereotyping in Latent Space

FORMA reduces trauma-category stereotyping while increasing cross-model convergence on the same persona. The effect is supported by embedding analyses and inter-rater agreement patterns, with visualization-based evidence requiring cautious interpretation.

  • Latent-space evidence: Zero-shot vignettes form tight trauma-category clusters, whereas formulation conditions disperse categories across the embedding layout.This indicates greater reliance on persona-specific content under no-formulation and full conditions, but the t-SNE display is qualitative.
  • Cross-model convergence: The full condition increases within-persona cross-model similarity over zero-shot (∆µ = +0.055, Cohen’s dz = +1.52).All three pairwise contrasts are highly significant, with the full-versus-zero-shot contrast largest.
  • Robustness: The same tier-by-condition interaction reproduces with BGE, while conditional and tier gradients remain identical to MPNet.Absolute cluster cohesion is lower with BGE, which is less sensitive to trauma-category content.
  • Cross-model convergence: Mean within-persona similarity rises from µzero-shot = 0.737 to µfull = 0.792, while the full distribution is narrower.The reported pattern is interpreted as a shared content scaffold across generation models.
  • Inter-rater agreement: Multi-rater agreement is highest on Ehlers & Clark components in full conditions and collapses in zero-shot, especially for Threat and Strategies.All-5 agreement falls from 0.72 to 0.10 for Threat and from 0.88 to 0.11 for Strategies.

Q Mean Expert Rating per Metric and Condition

Clinical experts rate formulation-supported vignettes higher than zero-shot alternatives, with the strongest condition effects on Ehlers & Clark coverage and selected narrative-quality dimensions.

  • Perceived quality: Importance has the largest ordinal full-versus-zero-shot contrast (dz = +1.98), followed by grounded (+0.87), clarity (+0.81), narrative (+0.75), and relevance (+0.70).All three pairwise ordinal contrasts are significant after correction.
  • Ehlers & Clark coverage: Threat shows the largest Ehlers & Clark expert-rating contrast, with Cohen’s dz = +1.58 for full versus zero-shot.Other EC effects are Memory +0.71, Appraisals +0.88, and Strategies +0.76; Triggers is non-significant.
  • Ehlers & Clark coverage: Full-condition expert ratings show near-ceiling Ehlers & Clark coverage, while zero-shot ratings are lowest for Threat, Memory, Strategies, and Appraisals.The corresponding zero-shot values are Threat 0.19, Memory 0.53, Strategies 0.44, and Appraisals 0.40; full values range from 0.85–0.91.
  • Coverage comparison: DSM-5 coverage remains near ceiling across conditions, whereas the formulation effect concentrates on Ehlers & Clark items.The within-panel contrast separates already-saturated diagnostic coverage from formulation-sensitive cognitive components.
  • Perceived quality: The ordinal ratings follow a full > no-formulation > zero-shot gradient, with importance and grounded showing the largest gaps.This gradient is clearer for human experts than for the scaled LLM judge.

R Scaled-Judge Mean Rating per Metric and Condition

The scaled LLM judge detects the formulation effect most reliably on Ehlers & Clark content and selected quality dimensions, but its surface-fluency sensitivity weakens or reverses some expert-aligned ratings.

  • Rater divergence: Zero-shot slightly exceeds full on Clarity (d = −0.08), illustrating the judge’s preference for unconstrained prose over structured clinical content.Human experts show the opposite direction on comparable ordinal ratings.
  • Ordinal quality: Full exceeds zero-shot on Importance and Narrative, with d = +0.52 and d = +0.76, respectively.These are the ordinal metrics most closely tied to clinical content, and the judge agrees with experts on direction.
  • Ehlers & Clark coverage: The judge’s Ehlers & Clark effects are positive for Appraisals (d = +0.55), Strategies (+0.76), and Memory (+0.26) in full versus zero-shot.Triggers slightly favors zero-shot (d = −0.07), consistent with saturation.
  • Rater divergence: The human-expert condition gradient is stronger than the scaled judge’s gradient on ordinal quality items.The divergence reflects different sensitivities: experts read for clinical meaningfulness, whereas the judge responds more to surface properties.
  • User-study context: All nine quality contrasts and all three authorship contrasts in the user study are significant at p < 10^-15.Figure 21 summarizes these contrasts using mixed-model quality differences and GEE authorship odds ratios.

X Algorithmic-Fairness Metrics by Condition

FORMA reduces demographic disparity in perceived quality across conditions, while EC-coverage parity is generally maintained but can worsen for trauma categories because content fit differs across personas.

  • Perceived-quality disparity: Perceived-quality SPD falls 1.5–7× from zero-shot to full across gender, ethnicity, age, and trauma category.The trauma-category disparity declines from SPD = 0.092 in zero-shot to 0.013 in full.
  • Coverage parity: Trauma-category EC disparity widens under full, with SPD rising from 0.053 to 0.094 and DIR falling from 0.92 to 0.89.The paper attributes this pattern to different natural mappings between trauma types and the specified EC components.
  • Demographic breakdown: Figures 24–25 show small within-condition demographic differences while preserving the condition gradient across gender and ethnicity.Group means are tightly aligned within each demographic level in the displayed analyses.
  • Coverage parity: EC coverage is invariant across demographics in both the scaled-judge set and clinician user study, with all omnibus tests p > 0.13.The user study also finds no detectable demographic effect on quality or AI authorship, with all p > 0.18.
  • Perceived-quality disparity: Trauma-category quality disparity changes from dmax = −0.49 in zero-shot to +0.07 in full, while DIR rises from 0.97 to 0.99.The interaction is highly significant, but the full-condition effect is negligible in magnitude.
Loading 2608.29995v1…