Source-linked AI summary
ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation
Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, Meliha Yetisgen
TL;DR
Clinical dialogue-to-note generation lacks sufficiently large public datasets because encounters are difficult to record and share ethically. The paper introduces ACI-BENCH, a corpus spanning multiple note-generation settings and speech-transcription conditions, and benchmarks common summarization approaches. The corpus is publicly positioned for systematic evaluation, while reported comparisons show strong performance from both large language and retrieval-based systems.
Problem
Public dialogue–note datasets are insufficient for training and evaluating clinical note-generation systems because medical encounters are rarely recorded and are difficult to share ethically.
Method
The paper constructs ACI-BENCH from multiple doctor–patient interaction modes, transcription conditions, cleaned note content, and structured note divisions, then evaluates baseline note-generation methods.
Results
GPT-4 achieved a MEDCON score of 57.78, while retrieval-based baselines reached ROUGE-L F1 scores of 40.47 and 38.20.
Takeaways & Limitations
ACI-BENCH provides a public benchmark for comparing clinical dialogue-to-note generation systems and studying their strengths and weaknesses.
Takeaways & Limitations
The dataset is small, synthetic, and produced at one institution, so it may not represent the full range of health topics, speech variations, or note formats.
Abstract
from arXiv · showhide
Recent immense breakthroughs in generative models such as in GPT4 have precipitated re-imagined ubiquitous usage of these models in all applications. One area that can benefit by improvements in artificial intelligence (AI) is healthcare. The note generation task from doctor-patient encounters, and its associated electronic medical record documentation, is one of the most arduous time-consuming tasks for physicians. It is also a natural prime potential beneficiary to advances in generative models. However with such advances, benchmarking is more critical than ever. Whether studying model weaknesses or developing new evaluation metrics, shared open datasets are an imperative part of understanding the current state-of-the-art. Unfortunately as clinic encounter conversations are not routinely recorded and are difficult to ethically share due to patient confidentiality, there are no sufficiently large clinic dialogue-note datasets to benchmark this task. Here we present the Ambient Clinical Intelligence Benchmark (ACI-BENCH) corpus, the largest dataset to date tackling the problem of AI-assisted note generation from visit dialogue. We also present the benchmark performances of several common state-of-the-art approaches.
1 Background & Summary
Clinical note generation could reduce documentation burden, but progress is limited by the scarcity of public encounter dialogue–note data. ACI-BENCH addresses this gap with a publicly available corpus covering multiple assisted note-generation settings and benchmarking workflows.
- Electronic health records have increased clinicians’ documentation burden, with concerns about delayed, incomplete, and difficult-to-read notes.
- Automatic summarization can capture and draft clinical notes for clinician review, enabled by advances in speech-to-text, electronic records, and transformer models.
- Publicly shareable dialogue–note data is scarce because encounters are rarely recorded and medical information is highly sensitive, limiting systematic comparison of models.
- ACI-BENCH provides a corpus spanning virtual-assistant, scribe-directed, and natural doctor–patient conversations, with human-transcription, ASR, and corrected-ASR variants.
- The paper details corpus construction, statistics, validation against real data, and baseline summarization evaluations.
2 Methods
The corpus models several realistic ways clinicians generate notes, cleans unsupported content, and organizes notes into contiguous divisions for training and evaluation. Its design also addresses variable note structures and long sequences that challenge standard evaluation models.
- Data Creation: ACI-BENCH represents virtual-assistant, virtual-scribe, and ambient clinical-intelligence encounters, covering explicit commands, scribe-directed speech, and natural doctor–patient conversation.
- Data Cleaning and Annotation: Medical experts created or reviewed transcripts and notes, while unsupported note text was identified and removed through annotation and automatic processing.
- Data Cleaning and Annotation: The dataset omits linkages between conversational input and independent EHR data such as vitals, survey responses, orders, and diagnosis codes.
- Note Division Definition: Because clinical notes lack a universal structure, the corpus groups content into contiguous SUBJECTIVE, OBJECTIVE_EXAM, OBJECTIVE_RESULTS, and ASSESSMENT_AND_PLAN divisions.
- Note Division Definition: Concatenated gold and system summaries can exceed BERT’s 512-subtoken limit, complicating fair embedding-based evaluation.
- Note Division Definition: Sub-note divisions support shorter sequence-to-sequence inputs and finer-grained evaluation than full-note processing.
3 Data Records
The corpus provides organized challenge and experiment data, with metadata and source-target files linked through encounter identifiers or dataset-specific ids.
- Challenge data are stored in the challenge_data folder, while ASR experiment data are stored in the src_experiment_data folder.
- Each data split contains an associated metadata file and data file.
- Metadata files record dataset, encounter, participant, and complaint fields, including dataset name, identifiers, names, gender, chief complaint, and secondary complaints.
- Source-target files contain transcript and note text alongside dataset name and id or encounter_id.
- Use encounter_id for challenge data and id for source experiment data when joining source-target files with metadata.
4 Technical Validation
ACI-BENCH was validated against real encounters and used to compare retrieval, pretrained, fine-tuned, and prompted models for clinical note generation. Results favored division-based generation and showed trade-offs between lexical overlap, clinical concept coverage, and note structure.
- Comparison with real data: ACI-BENCH contained ten times more QA-labeled transcript sentences than dictation, suggesting a corpus skew toward question-answer and conversational statement content.QA versus dictation proportions were 43% versus 4% in ACI-BENCH, compared with 15% versus 8% in CONSULT.
- Baseline experiments: BART+FTSAMSum (Division) achieved the strongest full-note ROUGE performance, while GPT-4 achieved the highest MEDCON score among the reported models.BART+FTSAMSum (Division) reached 53.46 ROUGE-1, 25.08 ROUGE-2, and 48.62 ROUGE-L; GPT-4 reached a MEDCON score of 57.78 with lower structured-note performance.
- Baseline experiments: Different models led across note divisions: BART+FTSAMSum (Division) scored 51.08 in SUBJECTIVE, while Text-davinci-003 led the three objective and planning divisions.Text-davinci-003 scores were 55.30, 48.90, and 46.19 for OBJECTIVE_EXAM, OBJECTIVE_RESULTS, and ASSESSMENT_AND_PLAN, respectively.
- Baseline experiments: Division-based generation significantly outperformed full-note generation for pretrained BART and LED models, improving full-note ROUGE and MEDCON evaluations by 1 to 14 points.For BART-based models, division-based average scores increased by around 5 to 40 points; LED gains were smaller but still present.
- Baseline experiments: Retrieval-based models achieved the best BERTScore in several objective and planning divisions but often had poor MEDCON because patient-specific conditions were less accurate.Their advantage was strongest for OBJECTIVE_EXAM, OBJECTIVE_RESULTS, and ASSESSMENT_AND_PLAN, where standard phrasing or templates aided retrieval.
- Baseline experiments: ASR versus human-transcription inputs had no remarkable effect on virtscribe note-generation performance, and further ASR fine-tuning produced only small score changes.For example, ROUGE-L changed from 43.98 with human transcripts to 41.74 with ASR input, while ASR fine-tuning reached 43.82.
5 Usage Notes
The dataset is intended for benchmarking clinician-patient dialogue summarization, with processing resources provided, but its scope is limited by synthetic single-institution data and missing demographic modeling.
- Processing instructions and evaluation examples are available through the Figshare README and associated GitHub code.
- The data is small, synthetically produced by medical annotators or patient actors, and drawn from a single institution.
- The dataset may not statistically represent all health topics, speech variations, and note-format variations found in the real world.
- The dataset is intended for benchmarking methods related to clinician-patient dialogue summarization.
- The data should not be used to train models for medical diagnosis.
- The dataset does not model demographic information beyond roughly equal gender balance across the full dataset.
7 Author contributions statement
The author contributions covered annotation guidelines, supervision, baseline experiments, corpus analysis, source-data acquisition, and manuscript drafting and revision.
- Contributors developed annotation guidelines, supervised annotation, ran or advised on baseline experiments, analyzed corpus data, and drafted or revised the manuscript.
9 Appendix
The appendix documents post-processing, example outputs, model settings, and results across test sets and note divisions. It highlights strong division-based BART and GPT-4 performance, while showing division-specific challenges and occasional differences across metrics and samples.
- Post-processing rules were added so the rule-based section algorithm could correctly split generated notes into divisions.
- GPT-4 correctly identified a patient’s correction from right knee pain to left knee pain, unlike the compared BART output.
- The appendix provides example model outputs with detected UMLS concepts and hyper-parameters for BART- and LED-based baselines.
- BART- and LED-based models used shorter maximum target lengths because longer target lengths produced poorer generation results in experiments.
- Division-based BART models and GPT-4 were among the most competitive models for full-note Rouge scores across reported test sets.
- The appendix reports that the strongest model could vary by metric, division, test set, and sample, including MEDCON differences involving Text-davinci-003 and transcript-copy baselines.
- LED models struggled with OBJECTIVE_EXAM, while OpenAI models performed best on OBJECTIVE_RESULTS in the reported test-set summaries.
- For ASSESSMENT_AND_PLAN, BART division models and OpenAI models were reported as the best-performing approaches in the summarized test sets.