Source-linked AI summary
Can large language models unlock discrete data in ophthalmic diagnostic reports?
Umair A. Zaidi, An-Lun Wu, Wei-Chun Lin, Thomas S. Hwang, Michelle R. Hribar
TL;DR
Ophthalmic diagnostic PDFs contain clinically important data that are often not machine-readable, and their varied layouts complicate extraction. This study evaluates two GPT-4o prompting pipelines for structuring those data; in the pilot, processing time fell by approximately 92% while accuracy remained high against reconciled manual ground truth.
Problem
Discrete data in ophthalmic diagnostic reports are often trapped in human-readable PDFs, and heterogeneous layouts complicate automated extraction and aggregation.
Source: Introduction · p. 1Method
The study evaluated two LLM prompting strategies for extracting structured data from ophthalmic diagnostic reports.
Source: Introduction · p. 2Results
Approximately 92% reduction in mean per-report processing time accompanied high accuracy against reconciled consensus ground truth in this pilot setting.
Source: Discussion · p. 7Takeaways & Limitations
Similar pipelines may be adaptable to additional report types with limited development overhead, although this requires further evaluation.
Source: Discussion · p. 8The pilot included only 20 de-identified, machine-readable reports from four report types at one institution and evaluated only GPT-4o and two prompting strategies.
Source: Discussion · p. 8
Abstract
from arXiv · showhide
Objective: To assess the accuracy and efficiency of a large language model (LLM) using two prompt strategies to extract structured data from ophthalmic diagnostic PDF reports. Methods: Twenty deidentified reports across four types (Visual Field, OCT Glaucoma Overview, OCT retinal nerve fiber layer Single Exam, and OCT Thickness Map; n = 5 each) were processed using two GPT-4o-assisted pipelines and compared with a reconciled manual ground truth. Schema-Constrained used Structured Output mode with a predefined JSON Schema; Prompt-Only used a detailed instruction prompt followed by Python conversion to JSON. Outcomes were value accuracy, formatting accuracy, and extraction time. Results: Schema-Constrained value accuracy was 100.00% for Visual Field and RNFL Single Exam, 97.45% for Glaucoma Overview, and 98.00% for Thickness Map; Prompt-Only achieved 100.00% across all four report types. Formatting accuracy was 100.00% for Schema-Constrained across all report types and 100.00% for Prompt-Only except RNFL Single Exam (90.14%). Mean extraction time was 56.51 s per report for manual review versus 5.04 s for Schema-Constrained and 4.70 s for Prompt-Only, an approximately 92% reduction. Conclusions: In this small proof-of-concept dataset, general-purpose LLM-assisted pipelines extracted structured data from ophthalmic diagnostic PDFs with high accuracy and substantially reduced processing time. Prompt-Only achieved the highest value accuracy, while Schema-Constrained produced schema-compliant output with 100% formatting accuracy. These complementary strengths support further evaluation of hybrid, validation-aware workflows for research and clinical data abstraction.
Introduction
Ophthalmic diagnostic PDFs contain clinically important measurements but are often not machine-readable, and varied vendor layouts challenge automated extraction. This study evaluates two LLM prompting strategies for extracting structured data from these reports.
Clinical context
- Diagnostic measurements support clinical decisions and research, including glaucoma diagnosis and monitoring and retinal disease assessment.Visual fields and OCT provide quantitative measures; OCT biomarkers include central subfield and nerve fiber layer thickness.Source: Introduction · p. 1
Data-access problem
- PDF reports often keep discrete data in human-readable form, while proprietary formats and layout variation complicate machine extraction and aggregation.Rule-based and OCR approaches can be brittle to changes across devices and over time, leaving data less available for EHR integration and downstream use.Source: Introduction · pp. 1–2
Research gap
- Prior ophthalmic NLP has largely targeted narrative clinical notes, while extracting precise field-value associations from dense, device-generated diagnostic PDFs has received little attention.These reports organize numeric fields by anatomic sector and laterality in fixed visual layouts.Source: Introduction · p. 2
Study aim
- LLM-assisted extraction from ophthalmic diagnostic reports is evaluated with two prompting strategies, focusing on accuracy and consistency of extracted values.The study frames the work as a proof-of-concept evaluation.Source: Introduction · p. 2
Methods
The study compared two GPT-4o workflows on 20 de-identified ophthalmic reports, using reconciled manual transcription as ground truth. The workflows differed in how they structured outputs, while sharing the same extracted PDF text and evaluation measures.
Study dataset
- Twenty de-identified reports covered four report types (n = 5 each); only PDFs with machine-readable embedded text were included, and scanned-image reports were excluded.Reports were de-identified and the process was verified by a second reviewer.Source: Methods · p. 2
Manual ground truth
- A trained assistant transcribed each report, two reviewers independently checked fields against the PDF, and discrepancies were resolved by consensus to establish ground truth.Manual extraction time was measured with a stopwatch.Source: Methods · p. 3
Shared input
- Both pipelines received the same text extracted from embedded PDF text layers; no OCR or pre-input normalization was applied, and absent fields were returned as null.Text extraction used pdfplumber and PyMuPDF; headers, footers, and watermarks were retained.Source: Methods · p. 3
Schema-Constrained Parsing
- Schema-Constrained Parsing used GPT-4o Structured Output mode with a predefined JSON Schema, producing schema-compliant JSON without additional formatting or text-to-JSON conversion.Figure 2 illustrates the minimal prompt and schema configuration; Figure 3 shows an RNFL Single Exam output example.Source: Methods · p. 4, Figure 2 · p. 5, Figure 3 · p. 5
Prompt-Only Parsing
- Prompt-Only Parsing used a detailed instruction prompt and returned key-value text that required post-processing into JSON.Figures 4 and 5 show the prompt configuration and an RNFL Single Exam text output before conversion.Source: Methods · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
Evaluation
- Value accuracy assessed value matches, formatting accuracy assessed key and value formatting, and extraction time measured the total time to parse each report.Each of the 20 reports was processed five times with both pipelines; GPT-4o was accessed through the OpenAI API without fine-tuning.Source: Methods · p. 6
Results
Prompt-Only achieved perfect value accuracy across the report types, while Schema-Constrained maintained perfect formatting accuracy. Both automated pipelines reduced mean extraction time by about 92% versus manual review.
Results
- Twenty de-identified reports—five each from four report types—were processed five times per method.The report types were Visual Field, OCT Glaucoma Overview, OCT RNFL Single Exam, and OCT Thickness Map.Source: Results · p. 6
- Prompt-Only achieved 100.00% ± 0.00% value accuracy across all report types; Schema-Constrained ranged from 97.45% ± 5.69% to 100.00% ± 0.00%.Schema-Constrained accuracy was 97.45% ± 5.69% for Glaucoma Overview and 98.00% ± 4.47% for Thickness Map, and 100.00% ± 0.00% for Visual Field and RNFL Single Exam.Source: Table 2 · p. 6, Results · p. 6
- 100.00% ± 0.00% formatting accuracy was achieved by Schema-Constrained across report types; Prompt-Only reached 90.14% ± 5.98% for RNFL Single Exam.The RNFL formatting errors involved omitting a required space between a section heading and its value.Source: Results · pp. 6–7, Table 3 · p. 7
- ~92% lower mean extraction time than manual review was achieved by both automated pipelines: 5.04 s per report for Schema-Constrained and 4.70 s for Prompt-Only, versus 56.51 s manually.Manual extraction ranged from 11.09 s to 82.34 s per report.Source: Table 4 · p. 7, Results · p. 7
Discussion
The pilot found that a general-purpose LLM extracted structured ophthalmic PDF data accurately and quickly across four tested report formats. The discussion presents portability and machine-readable outputs as potential routes to broader clinical and research use, while noting limits on generalizability.
Discussion
- Approximately 92% lower mean per-report processing time accompanied high accuracy against the reconciled consensus ground truth.The authors describe these results as suggesting a viable alternative to manual transcription in this pilot setting.Source: Discussion · p. 7
- Schema-Constrained formatting accuracy was 100% across all four report types, while Prompt-Only reached 100% on three and 90.14% on RNFL Single Exam.The system was evaluated on Visual Field, OCT Glaucoma Overview, OCT RNFL Single Exam, and OCT Thickness Map reports.Source: Discussion · p. 7
- The pipeline’s design remained unchanged across report formats and required no model fine-tuning or architecture changes, suggesting adaptation could take minor modifications.The authors describe the pipeline as potentially deployable across heterogeneous imaging devices and evolving software versions.Source: Discussion · pp. 7–8
- Portability may mitigate the brittleness of rule-based approaches when layouts or phrasing vary, a practical concern across devices and software versions [7,8,21].The discussion connects this potential to real-world ophthalmology environments where device and software variation occurs across and within institutions.Source: Discussion · p. 8
- Structured outputs could facilitate exchange among EHRs, registries, and research databases, and support aggregation of heterogeneous imaging into analyzable cohorts [4–7,23,24].The authors describe possible applications including interoperability standards mapping, retrospective registry construction, and multisite collaboration.Source: Discussion · p. 8
- Generalizability is limited by the 20-report pilot, its single-institution and four-format scope, and exclusion of scanned image PDFs and handwritten annotations.Only GPT-4o and two prompting strategies were evaluated; performance may differ across models, institutions, vendors, and report formats.Source: Discussion · p. 8
Funding
The study received support from federal, endowed, and departmental sources.
Funding
- Support included NIH awards R01EY030929, R01EY013178, R50EY038047, R21EY037312, T15LM007088, P30EY010572, and P30EY001792.P30EY010572 is identified as an OHSU core grant and P30EY001792 as a UIC core grant.Source: Funding · p. 8
- Additional support came from the Malcolm M. Marquis, MD Endowed Fund for Innovation and unrestricted departmental grants to the Casey Eye Institute and the University of Illinois Chicago.The sponsors and funding organizations had no role in the research design or conduct.Source: Funding · pp. 8–9
Conflict of Interest
The authors report no conflicting relationships.
Conflict of Interest
- No conflicting relationship exists for any author.Source: Conflict of Interest · p. 9