Source-linked AI summary
Symphony for Text Generation: Benchmarking Clinical Note Generation
Daniel Varab, Victor Petrén Bach Hansen, Asbjørn W. Helge, Kevin Pelgrims, Mathias Baltzersen, Adrian Young-San Roessler, Vanessa Klungtvedt, Maximilian Brand, Lasse Krogsbøll, Henrik Cullen, Lars Maaløe
TL;DR
Clinical note quality remains under-reported and difficult to compare as ambient documentation systems gain adoption. The paper introduces a controlled multilingual benchmark and evaluation framework, finding that Corti performs strongly across datasets and can be configured for different documentation priorities.
Problem
Clinical note quality remains under-reported, and cross-vendor comparisons vary in scale and control while omissions can escape evaluations focused only on note content.
Source: §1 · p. 2Method
The study pairs MedConv and ACI-BENCH with controlled comparisons of three systems, using entailment metrics and LLM pairwise judgments across eight PDSQI-9 dimensions.
Source: §1 · p. 2, §2.1 · p. 3, §2.4 · p. 5Results
Corti achieves the highest completeness and groundedness across four datasets, runs fastest, and is preferred over both comparators on average across PDSQI dimensions.
Source: §4 · p. 8Takeaways & Limitations
The findings support evaluating ambient documentation as configurable systems whose preferred quality profile can depend on clinical setting and workflow.
Source: §4 · p. 8Clinician validation measured ratification of the LLM judge’s declared winner rather than independent preference, so reported agreement should be read as an upper bound on concordance.
Source: H Limitations · p. 22
Abstract
from arXiv · showhide
Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedConv, a multilingual dataset of 300 clinical encounters in English, Danish, and German, and use it alongside the Ambient Clinical Intelligence benchmark (ACI-BENCH) to compare Corti, a clinical AI platform, with two leading, accessible ambient scribe software applications built on general-purpose AI. We present a controlled clinical evaluation framework that combines entailment metrics with LLM-judged pairwise comparisons across eight dimensions adopted from PDSQI-9. Results show that Corti's API-based text-generation infrastructure is on par with or outperforms leading commercial scribes. We further show that Corti's configurable API provides the flexibility necessary to fine-tune quality dimensions for specific documentation use cases. We present the evaluation methodology and release a dataset to support future reproducible comparison of ambient documentation systems.
1 Introduction
Ambient documentation is increasingly adopted, but clinical note quality remains under-reported and difficult to compare systematically. The paper introduces a controlled evaluation methodology and dataset to compare systems and assess configurable documentation quality.
1 Introduction
- Clinical note quality remains under-reported despite reported workflow benefits, and cross-vendor comparisons vary in scale and control of system inputs and configurations.Omissions are especially difficult to assess because evaluators focused on what a note contains can miss what it leaves out.Source: §1 · p. 2
- BLEU, METEOR, and ROUGE are described as poorly suited to clinical text, motivating semantic, factual, clinician, and LLM-based evaluation approaches.The passage describes these as limitations of surface-level measures and summarizes recent approaches to evaluation.Source: §1 · p. 2
- Qualitative surveys and anecdotes can conflate underlying AI performance with workflow integration, interface design, or pilot clinicians’ preferences.The paper frames this as a risk for teams building ambient scribes and health systems selecting them.Source: §1 · p. 2
- The paper presents a methodology and public dataset, showing a purpose-built platform matches or outperforms commercial scribes while allowing tailoring to clinician preferences.The stated goal is to support tailoring documentation to workflow needs and quantifying resulting performance.Source: §1 · p. 2
- The evaluation combines entailment measures for groundedness, completeness, and conciseness with LLM pairwise judgments across PDSQI-9 dimensions, with clinicians validating a subset.MedConv contains 300 synthetic scenarios with matched transcript-note pairs in English, Danish, and German; the benchmark also evaluates Heidi, Tandem Health, and Corti on MedConv and ACI-BENCH.Source: §1 · p. 2
2 Methodology
The methodology uses paired, controlled note generation across two datasets and three systems, then assesses content fidelity and clinician-oriented quality. Clinician ratings generally endorsed the pairwise judge, with the reported validation confined to a targeted sample.
2.1 Experimental Design
- Systems receive identical pre-transcribed inputs under harmonized four-section SOAP templates and default settings; the study excludes speech-recognition errors and evaluates SOAP notes only.Evaluation is at encounter level and combines reference-based and reference-free coverage measures with paired quality comparisons.Source: §2.1 · p. 3
2.2 Evaluation Datasets
- The benchmark combines ACI-BENCH with MedConv, which adds medical domains, consultation settings, and languages; MedConv contains 300 transcript-note pairs across English, Danish, and German.MedConv spans 15+ specialties and uses synthetic consultation transcripts paired with clinician-validated notes; ACI-BENCH’s aci subset is used as the closest match to ambient scribing [Yim et al., 2023].Source: §2.2 · p. 3
2.3 Systems Under Review
- The comparison includes Heidi and Tandem Health applications and Corti’s Symphony text-generation API, with templates and output-affecting settings harmonized or fixed to vendor defaults.For the commercial applications, manual generation limited MedConv sampling to one note per encounter, versus five notes per encounter on ACI-BENCH; Corti was sampled five times per encounter across datasets.Source: §2.3 · p. 4
2.4 Evaluation Measures
- Three entailment measures assess completeness, conciseness, and groundedness, while pairwise LLM judgments extend PDSQI-9 across eight dimensions, with order-swapped comparisons and disagreements recorded as ties.The study also reports runtime, cluster-bootstrap confidence intervals, and judge-model sensitivity; dimension-level results are descriptive because no multiplicity correction is applied.Source: §2.4 · p. 5
2.5 Clinical Validation of the Pairwise Evaluation
- Three clinicians separately reviewed blinded pairwise comparisons from 10 ACI-BENCH notes, with note order randomized and both position-swapped judgment traces shown.The clinician checks provide a targeted validation of the LLM pairwise judgments.Source: §2.5 · p. 5
- 207 of 240 clinician assessments endorsed the judge’s verdict (86.3%; Gwet’s AC1 = 0.694), and clinicians agreed by majority in 75 of 80 case-dimension comparisons.Agreement was highest for accuracy, thoroughness, and usefulness, and lowest for synthesis (AC1 = 0.080).Source: §2.5 · p. 5
3 Results
Across four datasets, Corti produced the most complete notes, while systems scored similarly on groundedness and conciseness. Pairwise preferences exposed dimension-specific trade-offs, and targeted general-practice instructions improved Corti’s succinctness preference.
3.1 Overall performance evaluation of the three systems
- Corti produced the most complete notes across four datasets; completeness was the clearest differentiator, while groundedness and conciseness were generally similar.Completeness scores ranged from 62–77%, while groundedness was consistently 90–98%; groundedness and conciseness were generally within two points across systems.Source: §3.1 · p. 6
- Corti had the lowest mean latency across datasets at 5.9 to 9.7 seconds per note; Heidi took 2.0 to 2.7 times as long and Tandem Health 1.9 to 2.2 times as long.Source: Table 3 · p. 6, §3.1 · p. 6
3.2 Granular evaluation on clinical dimensions
- Corti won overall 63% against Heidi and 66% against Tandem Health, leading on accuracy, thoroughness, and usefulness but trailing on succinctness.Its largest advantage was thoroughness; comparator-dependent differences also appeared in synthesis, comprehensibility, and organization.Source: Figure 2 · p. 7, §3.2 · p. 7
3.3 Targeted intervention to improve on succinctness
- Against Heidi, organization fell from 46.1% to 35.8% and comprehensibility from 46.0% to 40.5%; against Tandem Health, synthesis shifted from 51.7% to 48.1%.These changes accompanied the succinctness gains from the general-practice intervention.Source: §3.3 · p. 7
- After general-practice instructions, Corti’s succinctness preference rose from 16.9% to 65.3% against Heidi and 18.2% to 59.7% against Tandem Health.Only two narrowly scoped instruction fields changed while the upstream base section remained unchanged; thoroughness declined but stayed favorable above 50%, and other dimensions shifted unevenly.Source: §3.3 · pp. 7–8, Figure 3 · p. 8
4 Conclusion
The evaluation framework finds that Corti leads the comparators on completeness and groundedness and is preferred on average, while configurable templates can shift documentation style for different clinical contexts.
4 Conclusion
- Corti achieved the highest completeness and groundedness across four datasets, the fastest generation runtime, and average preference over both comparators across PDSQI dimensions.The conclusion also reports that a small surgical template intervention shifted succinctness while preserving an above-neutral preference for thoroughness.Source: §4 · p. 8
- Completeness may matter more in complex encounters, while brevity may reduce review burden in high-volume, lower-complexity settings.The paper argues ambient documentation should be evaluated as configurable systems whose quality profile depends on clinical setting and workflow.Source: §4 · p. 8, A Related Work · p. 13
A Related Work
- Earlier comparisons found substantial variability across scribes, with omissions prominent and input-modality or template differences limiting pipeline-level conclusions.Omission-focused evaluation and PDSQI-9 provide complementary approaches to assessing generated notes [Croxford et al., 2025a].Source: A Related Work · p. 13
B Symphony for Text Generation
- Symphony accepts varied clinical context and uses FactsR-extracted facts with transcripts in the experimental note-generation pipeline.Guided synthesis uses templates to produce schema-controlled documents, while reusable typed sections separate content, style, and output-format controls for independent tuning.Source: B Symphony for Text Generation · p. 14
C.1 Template Configuration Details
- Template architecture and configuration transparency are compared across the evaluated systems.The supplied table caption establishes the comparison's scope but does not specify the systems' relative differences.Source: B Symphony for Text Generation · p. 14
C System configurations
- The benchmark compares three systems using pre-transcribed consultation content, keeping speech-to-text performance outside the comparison.The cited configuration tables cover system configurations and country-specific Heidi and Tandem SOAP templates; Corti template configurations are accessible through its Console or public API.Source: Table 4 · p. 15, Table 5 · p. 16, Table 6 · p. 16
D Template Architecture Differences
Table 8 compares template architecture and configuration transparency across the evaluated systems.
D Template Architecture Differences
- Table 8 covers template architecture and configuration transparency across the evaluated systems.The caption does not state a relative result or describe the table's row and column encodings.Source: Table 8 · p. 17
E.1 Entailment-based evaluation metrics
The evaluation frames semantic alignment as textual entailment: an LLM labels note hypotheses against segmented premises, and the resulting scores measure completeness, conciseness, or groundedness depending on the source document used.
E.1 Entailment-based evaluation metrics
- An LLM independently judges whether each segmented hypothesis is entailed by a segmented premise, citing supporting source sentences.Labels distinguish entailed, partially entailed, not entailed, and not applicable hypotheses.Source: §E.1 · p. 18
- Applicable hypothesis labels receive weights of 1 for entailed, 1/2 for partially entailed, and 0 for not entailed, then are averaged.Not-applicable hypotheses are excluded; a score of 1 is assigned when every hypothesis is not applicable.Source: §E.1 · p. 18
- Completeness uses generated notes as premises and reference-note segments as hypotheses, while conciseness reverses those roles.Groundedness uses the transcript as premise and generated-note segments as hypotheses; unsupported content is not necessarily a genuine fabrication because judge or segmentation errors can also contribute.Source: §E.1 · p. 18
- Completeness and conciseness function like recall and precision against the reference note, whereas groundedness measures support in the transcript.The directional metrics use different premise-and-hypothesis assignments rather than measuring a learned proxy of preference.Source: §E.1 · p. 18
E.2 Pairwise Evaluation
Pairwise LLM judgments compare two notes from the same transcript on PDSQI-9 dimensions, with conservative agreement rules and tie-adjusted preference scores summarizing the results.
E.2 Pairwise Evaluation
- An LLM compares two notes from the same transcript on eight PDSQI-9 dimensions, returning a preference or tie for each dimension.The Cited attribute is excluded because the evaluated systems do not produce provenance citations.Source: §E.2 · p. 19, Table 9 · p. 19
- A system receives the aggregate Overall preference only when it wins at least five of the eight dimensions; otherwise, the result is a tie.The authors describe this as a simple summary statistic, while noting that quality dimensions may not correlate linearly with preference.Source: §E.2 · p. 19
- Pairs are judged in both presentation orders, and a system is preferred only when both judgments select it; disagreement becomes a tie.This guards against counting position-sensitive judgments as wins but does not remove other model-judge biases.Source: §E.2 · p. 19
- The tie-adjusted preference score is neutral at 50, with higher values favoring the reviewed system and lower values favoring its comparator.The calculation uses adjudicated wins for each system and ties.Source: §E.2 · pp. 19–20
- The 95% confidence intervals come from 10,000 paired-note bootstrap replicates, resampling all dimension outcomes from an encounter together.For pooled datasets, resampling is performed separately within each dataset to preserve the original dataset sizes.Source: §E.2 · p. 20
E.3 Judge Sensitivity
Judge preferences were similar across GPT-5.4 and Opus 4.6, while Corti had the fastest runtime across systems and datasets.
E.3 Judge Sensitivity
- Both judge models generally preferred Corti, with little difference between model families; Opus 4.6 scored ties more often than GPT-5.4.The analysis reports low judge sensitivity for the pairwise comparison method.Source: §E.3 · p. 20
F System Latency Extended
- Corti had the fastest runtime across all four datasets, with lower medians, fewer outliers, and narrower interquartile ranges.Heidi and Tandem Health were measured through browser interfaces, so their latency distributions are indicative.Source: F System Latency Extended · p. 20
G Targeted clinician validation of the pairwise judge
Clinicians agreed with the LLM judge in most assessments, but agreement varied by dimension and clinicians described different reasons for preferring a note.
G Targeted clinician validation of the pairwise judge
- 207 of 240 assessments (86.3%) agreed with the LLM judgment, and a majority of clinicians supported it in 75 of 80 case-dimension comparisons (93.8%).Clinicians agreed most on accuracy, thoroughness, and usefulness, and least on synthesis.Source: G Targeted clinician validation of the… · p. 20
- Gwet’s AC1 was 0.694 overall, highest for accuracy (0.929), thoroughness (0.837), and usefulness (0.827), and lowest for synthesis (0.080).Nominal Krippendorff’s alpha was 0.020 overall; it was undefined for stigmatizing-language ratings because all ratings were identical.Source: G Targeted clinician validation of the… · pp. 20–21
- Clinicians described different drivers of preference, including succinctness, fluent thoroughness, and critical flaws that outweighed succinctness.The annotation task and interface were reported as clear overall, though dimensions varied in assessment difficulty.Source: G Targeted clinician validation of the… · p. 21
H Limitations
The comparisons have boundaries around prompt transparency, system-level configurations, validation design, and language localisation.
H Limitations
- Tandem Health’s section-level prompts were hidden, so output differences may reflect prompt engineering as well as pipeline or model capability.Its English comparison also used a template built from section headings because no suitable vendor-provided UK template was available.Source: H Limitations · pp. 21–22
- Because the systems’ models, versions, and inference configurations are undocumented and vary alongside prompts and infrastructure, differences apply to systems as configured, not individual components.The systems also use different underlying LLMs and cloud infrastructure.Source: H Limitations · p. 22
- Clinician validation asked whether annotators agreed with the revealed LLM verdict and reasoning, so agreement rates and Gwet’s AC1 are upper bounds on judge–clinician concordance.Clinicians did not record independent preferences for later comparison.Source: H Limitations · p. 22
- Heidi and Tandem Health’s style and length settings were held at their defaults, while Corti’s equivalent controls were specified in section prompts.These system-specific parameters are injected together with a template.Source: H Limitations · p. 23
- Language prompt localisation varied across systems and may affect output quality in non-English markets, particularly Danish.Heidi’s Danish templates used English prompts, while Corti configured per-locale variants.Source: H Limitations · p. 23
I Characterization of ungrounded statements
The preliminary groundedness analysis found different rates and patterns of flagged statements across systems, but its LLM-assigned classifications are indicative rather than a confirmed clinical error audit.
I Characterization of ungrounded statements
- Corti had 655 flagged statements (160 S3+), versus 1,242 (534 S3+) for Tandem Health and 1,286 (459 S3+) for Heidi; fully unsupported statements numbered 23, 149, and 83, respectively.Most flagged content was partially entailed, with a supported core claim but an unstated added detail.Source: I Characterization of ungrounded… · p. 23
- Specificity overreach accounted for 22% of Corti, 26% of Tandem Health, and 40% of Heidi flagged entries; fabrication from silence was highest for Tandem Health (195 entries).Heidi showed more terminology substitution (156 entries) than Corti (55 entries).Source: I Characterization of ungrounded… · p. 23
- Because an LLM assigned failure modes and severity without independent clinical rubric validation, the analysis is an indicative error characterization, not a confirmed clinical error audit.Table 11 presents examples of ungrounded statements by system and severity.Source: I Characterization of ungrounded… · p. 23, Table 11 · p. 24