Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development

Background & Summary

• The dataset pairs radiologist gaze with dictation audio and timestamped text, and adds anatomical bounding boxes for every image.The bounding boxes can be combined with gaze information for more meaningful analyses.Source: Background & Summary · p. 1
• Eye coordinates during a finding provide an approximate region of interest, offering a low-cost route to local annotations without changing routine radiology reading.The shared ground-truth bounding boxes allow researchers to study how accurately gaze marks finding locations.Source: Background & Summary · pp. 1–2
• Using eye gaze in deep neural network training can improve performance, as demonstrated in the paper’s experiments.The dataset is presented as a resource for improving machine-learning development in radiology, where explainability and missing local annotations are concerns.Source: Background & Summary · pp. 1–2
• Attention maps of gaze on normal and diseased radiographs reveal radiologists’ cognitive workflow patterns for AI developers to use when building models.Source: Background & Summary · p. 2

Methods

• The study combined MIMIC-CXR images with MIMIC-IV Emergency Department data, then collected gaze and dictated reports during a radiologist’s readings.The final data include raw gaze signals, audio, and corrected transcripts; the databases had been de-identified and their research use approved by both institutional review boards.Source: Methods · p. 2, Ethical Statement · p. 2

Inclusion and Exclusion Criteria

• Sampling selected 1,083 cases equally across Normal, Pneumonia, and CHF, prioritizing disease-feature examples across ages and sexes.The sampling goal was to support computer-vision algorithms that do not overly rely on age and sex biases.Source: Methods · p. 2, Inclusion and Exclusion Criteria · p. 3
• Pneumonia and CHF labels came from clinician-verified ICD-9 discharge diagnoses, while normal cases required no relevant ICD-9 or report-derived abnormal labels.The authors state that discharge diagnoses incorporate information beyond the CXR, which may not by itself be specific enough for diagnosing pneumonia or CHF.Source: Inclusion and Exclusion Criteria · p. 2

Preparation of Images

• Images were converted to normalized, resized, padded PNGs at 1920x1080, and random calibration images were used to measure gaze offset.The PNG conversion retained the original DICOM aspect ratio; images were prepared for the eye-tracking software and monitor resolution.Source: Preparation of Images · p. 3

Software and Hardware Setup

• A Gazepoint GP3 tracker and its software were installed in the radiologist’s routine work environment, with sessions capped at 30 images to limit fatigue and interruptions.Each session began with a nine-point calibration.Source: Software and Hardware Setup · p. 3

Radiology Reading

• The radiologist read randomized images in unstructured prose, without exam-indication details, to approximate routine reading while keeping interpretation based on image features.The resulting dataset supports research questions involving reads with and without patient clinical information.Source: Radiology Reading · p. 3

Spreadsheet Merging

• The team reviewed session videos for technical problems, then merged gaze and fixation spreadsheets and mapped screen coordinates back to original MIMIC image coordinates.Source: Data Post-Processing · pp. 3–4, Spreadsheet Merging · p. 4

Audio Extraction and Transcript Generation

• Dictation audio was extracted from session videos, transcribed with Google Speech-to-Text, and manually corrected by three team members with radiologist verification.The transcript stores timestamps alongside words.Source: Audio Extraction and Transcript… · p. 4

Segmentation Maps and Bounding Boxes for Anatomies

• The dataset adds four anatomical segmentation maps per image, automatically generated and then manually reviewed and corrected by the radiologist.The masks cover the left lung, right lung, mediastinum, and aortic knob.Source: Segmentation Maps and Bounding Boxes… · p. 4
• Seventeen anatomical bounding boxes were automatically extracted and corrected when needed; researchers can combine these supplemental annotations with gaze for segmentation and disease-localization work.The authors also use the boxes to analyze fixation patterns across condition pairs.Source: Segmentation Maps and Bounding Boxes… · p. 4

Data Records

• The release comprises a master spreadsheet, raw gaze and fixation spreadsheets, bounding-box coordinates, and per-image audio, transcript, and segmentation files.Source: Data Records · p. 4

Master Spreadsheet

• The master spreadsheet links each record to its MIMIC-CXR image and report and includes report-derived finding labels from two NLP pipelines.It also contains exam-indication sentences extracted from the original reports.Source: Master Spreadsheet · p. 5

Fixations and Eye Gaze Spreadsheets

• The raw gaze file stores one row per eye-tracker sample, while the fixation file stores one entry per fixation averaged from those samples until a saccade is detected.Source: Fixations and Eye Gaze Spreadsheets · p. 5

Bounding Boxes Spreadsheet

• The bounding-box spreadsheet names 17 frontal-CXR anatomical zones and records each box’s upper-left and lower-right coordinates.Some boxes may be absent when the anatomy is cut off in the image.Source: Bounding Boxes Spreadsheet · p. 5

Audio, Segmentation Maps and Transcripts

• Each case folder contains dictation audio, anatomy segmentation maps, and a transcript with full text and timestamped phrases.Phrase records include begin and end times; the report also points readers to a transcript JSON example.Source: Audio, Segmentation Maps and Transcripts · pp. 5–6

Validation of Eye Gaze Data

• Calibration errors averaged 0.0089 on X and 0.0504 on Y as percentages, or 17.0356 pixels on X and 54.3943 pixels on Y.The calibration estimate used the final resting fixation across 59 randomly presented calibration images.Source: Validation of Eye Gaze Data · p. 6

Validation of Transcripts

• Experts corrected both transcript error types; among 19,499 phrases, 1,065 contained multiple words in one timestamp and were counted as Type B errors.Three experts corrected the transcripts and the radiologist verified them; missing phrase timing was estimated manually.Source: Validation of Transcripts · p. 6

Statistical Analysis on Fixations

• Fixations were higher in both diseased conditions than normal (p < 0.01), concentrated in diagnostically relevant regions, and Pneumonia/CHF patterns were similar but statistically different.The reported examples include Left Hilar Structure, Left Lung, Cardiac Silhouette, and Upper Mediastinum.Source: Statistical Analysis on Fixations · pp. 6–7

Usage Notes

• The dataset is hosted on PhysioNet, but users must separately apply for access to MIMIC-CXR to obtain the images.The repository includes usage descriptions, source code, and scripts for reproducing the validation results.Source: Usage Notes · p. 7

Temporal Heatmaps Experiment

• The temporal model concatenates image and gaze representations before the final classification layer, encoding the sequence of fixation heatmaps with a bidirectional LSTM and self-attention.The baseline uses the image representation alone.Source: Temporal Heatmaps Experiment · p. 7
• Adding temporal eye-gaze information improved AUC by 4% over the image-only baseline for the Normal, CHF, and Pneumonia classification task.The experiment used temporal fixation heatmaps without preprocessing, filtering, or feature engineering.Source: Temporal Heatmaps Experiment · p. 7, Use of the Dataset in Machine Learning · p. 7

Static Heatmaps Experiment

• The U-Net jointly predicts disease class and static fixation heatmaps; its AUC was similar to baseline, while its probability maps resembled gaze maps more than Grad-CAM maps did.The authors present this as an interpretability use case for static gaze heatmaps.Source: Static Heatmaps Experiment · pp. 7–8
• The experiments omit audio and text modalities because of their complexity and the paper's dataset-description scope.The authors describe the demonstrations as starting points for exploring uses of the multimodal dataset.Source: Static Heatmaps Experiment · p. 8

• The dataset reflects one radiologist, so it lacks inter-observer variability.The authors aim to expand collection to multiple radiologists.Source: Limitations of study · p. 8
• PNG images prevented the radiologist from using windowing techniques during review.The PNGs were prepared using windowing information from the original DICOM images.Source: Limitations of study · p. 8
• The dataset contains only PA images, so analyses may have limited generalizability to AP images.PA views were selected for their higher image quality compared with AP views.Source: Limitations of study · p. 8

• Inclusion and exclusion criteria are applied to the MIMIC dataset.Source: §1 · p. 8
• Cases are sampled and images prepared for the eye-gaze experiment.Source: §1 · p. 8

• Dictation audio undergoes speech-to-text processing.Source: §2 · p. 8
• Eye-gaze coordinates are mapped to original image coordinates.Source: §2 · p. 8
• Temporal or static heatmaps and videos are generated from eye-gaze coordinates.Temporal and static heatmap images are used in machine-learning demonstrations.Source: §2 · p. 8

• Eye-gaze fixation quality is validated using calibration images.Source: §3 · p. 9
• The quality of transcribed dictations is validated.Source: §3 · p. 9
• T-tests compare eye-gaze fixations across anatomical structures and condition pairs.Source: §3 · p. 9
• Machine-learning experiments are described in the dataset-use section, with software requirements listed in the GitHub repository.Source: §3 · p. 9

Figures & Tables

• Table 1 is an online-only master spreadsheet documenting descriptive ICD-9 diagnosis names for ER admissions prompting CXR studies, with final diagnoses defining three classes in eye-gaze analysis and experiments.It also describes original-report abnormal finding labels from an internal IBM CXR report labeler, coded 0 for negative and 1 for positive.Source: Table 1 · p. 10, Figures & Tables · p. 11
• Table 2 covers raw eye-gaze samples and fixation-level records, including blink, pupil, saccade, and original-image coordinate fields.The gaze and fixation spreadsheets are distinct records; fixation entries are derived from eye-gaze samples.Source: Table 2 · p. 12, Figures & Tables · p. 13
• Tables 3 and Figures 13–17 present static-heatmap hyperparameters, temporal and static gaze model architectures and results, and qualitative comparisons with Grad-CAM.Figure 17 compares a CXR, baseline and U-Net encoder Grad-CAMs, a static gaze heatmap, and a U-Net probability map.Source: Table 3 · p. 13, Figure 13 · p. 18, Figure 14 · p. 18
• Figures 2–4 show image selection from MIMIC-IV-ED and MIMIC-CXR, calibration with overlaid fixation, and a sample video of gaze over a CXR.Figure 2 depicts the selection criteria; Figure 3 shows the calibration image and fixation overlay; Figure 4 shows an exported case video.Source: Figure 2 · p. 14, Figure 3 · p. 15, Figure 4 · p. 15
• Figures 5–6 and 11–12 depict anatomical masks, overlapping bounding boxes, fixation patterns by anatomy and condition, and pairwise p-values.Figure 5 labels left lung, right lung, aortic knob, and mediastinum; Figure 6 overlays 17 anatomical boxes on a CXR.Source: Figure 5 · p. 15, Figure 6 · p. 15, Figure 11 · p. 17
• Figures 9–10 and 18 provide examples of manual detection or transcript correction and a radiology transcript.Figure 10 contrasts a missed or incorrect transcript phrase with its manually corrected version.Source: Figure 9 · p. 16, Figure 10 · p. 17, Figure 18 · p. 21