Source-linked AI summary
Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation
Leila Khaertdinova, Anna Anikina, Claudia Mello-Thoms, Bulat Ibragimov
TL;DR
Expertise assessment in radiology remains limited in volumetric CT, where gaze-based differences can be task- and modality-dependent and existing approaches often omit 3D spatial context. This paper introduces a gaze-informed DINOv2 transformer integrating fixation patterns into attention and pooling, achieving ROC-AUC 0.91 and F1 0.86 on held-out CT sessions.
Problem
Existing gaze-based skill-assessment studies mainly use 2D views or video clips, leaving generalization to 3D CT settings unclear.
Method
A DINOv2-based transformer integrates per-patch radiologist gaze signals through learnable log-space attention bias and gaze-weighted pooling.
Results
ROC-AUC 0.91 and F1 0.86 were achieved on held-out CT sessions, with the proposed model performing best across almost all reported metrics.
Takeaways & Limitations
The findings suggest that visual search behavior incorporated into transformers may support objective, process-based radiologist expertise assessment and gaze-based training feedback.
Takeaways & Limitations
Future work must increase participation beyond five radiologists to support more extensive clinical validation.
Abstract
from arXiv · showhide
Accurate interpretation of volumetric CT requires efficient navigation of 3D image volumes and attention to diagnostically relevant regions. While eye-tracking has been widely studied in 2D medical imaging, its use for expertise assessment in CT settings remains limited. We propose a gaze-informed transformer framework for expertise classification in thoracic CT. Using a DINOv2 backbone, radiologist fixation patterns are integrated into volumetric feature learning through (1) a learnable log-space bias in self-attention and (2) gaze-weighted pooling of patch embeddings. We trained and evaluated our approach on 182 CT reading sessions from five radiologists with varying levels of experience. On a held-out test set, the model achieves an ROC-AUC of 0.91 and F1 score of 0.86, outperforming adapted methods. These findings suggest that incorporating visual search behavior into transformers may support objective, process-based expertise assessment in radiology. Code is available via https://github.com/leiluk1/GazeToSkill.
1 Introduction
Radiology expertise is reflected in visual search behavior, but gaze-based skill assessment has focused mainly on 2D views or video rather than volumetric CT. This work addresses that gap with a gaze-informed transformer for 3D CT expertise modeling.
- Objective expertise assessment can support training programs that promote expert-like strategies and accelerate skill acquisition.
- Experts and novices show consistent differences in radiological visual search behavior, measurable through spatial and temporal eye-tracking patterns.
- Most gaze-based skill-assessment studies use 2D views or video clips without explicitly modeling three-dimensional spatial context.
- The proposed framework integrates radiologist attention into volumetric CT representation learning using gaze-biased attention and gaze-weighted pooling.
2 Dataset
The study collected synchronized eye-tracking data during radiologists’ interpretation of lung CT scans, alongside scanpaths and nodule annotations. These data support modeling expertise-related gaze behavior in volumetric imaging.
- CT Dataset: Three expert and two novice radiologists analyzed 40 lung CT scans from the LIDC–IDRI dataset.The scans included 24 cancer cases and 16 non-cancer cases containing 47 lung nodules.
- Scanpaths: Expert and novice scanpaths show fixation trajectories over normalized time, with lung nodules marked by orange segmentation masks.
- Experimental Protocol: Eye-tracking was recorded during clinical workflow actions including zooming, window-level adjustment, and navigation across orthogonal planes.
- Gaze-to-CT Mapping: Raw gaze coordinates were synchronized with video frames, corrected for drift, and mapped to CT coordinates using viewer metadata and nodule locations.
- Data Preprocessing: The preprocessing pipeline converts CT patches and downsampled gaze heatmaps into tokens and per-patch gaze weights for the model.
3 Methods
The method injects gaze into a DINOv2-based volumetric transformer at both self-attention and output pooling stages. Gaze weights emphasize attended patches while preserving transformer-based representation learning and session-level classification.
- Input and Patch Embedding: Each CT slice is converted into pseudo-RGB input, partitioned into patches, and linearly projected into ViT tokens with a prepended classification token.
- Gaze Heatmap to Patch Weights: Gaze heatmaps are average-pooled to the patch grid, flattened, and ℓ1-normalized to produce per-patch gaze weights.
- Gaze-Bias Attention: At every transformer layer, a learnable additive log-space gaze bias modifies self-attention logits according to each key position’s gaze weight.
- Gaze-Bias Attention: The biased attention scales each key’s attention probability by w_j^α, amplifying heavily fixated patches and suppressing lightly viewed patches.
- Output Pooling and Classification: Final patch tokens are combined through gaze-weighted pooling, mean-aggregated across sampled slices, and passed to a two-layer MLP for novice-versus-expert logits.
4 Results
The evaluation used 182 gaze-recorded CT sessions, with held-out testing and stratified cross-validation addressing the expert-majority class imbalance. Results also compare backbone representations and gaze-integration configurations.
- Experimental setup: 182 samples remained after excluding incomplete recordings, with 115 training samples and 67 held-out test samples.The test set contained 39 expert and 28 novice sessions; radiologists C and E were held out.
- Experimental setup: Stratified 5-fold cross-validation addressed the approximately 70% expert class imbalance, with fold predictions averaged at inference.The final classification threshold was determined using Youden’s J on the test set.
- Implementation: The implementation trained with lung-windowed 518×518 slices, Adam optimization, 100 epochs, and validation ROC-AUC checkpoint selection.Each step sampled 8 random slices per CT volume with gradient accumulation over 4 steps, yielding a batch size of 4 sessions.
- Evaluation metrics: Table 1 compares methods on the held-out test set using ROC-AUC, F1, Sensitivity, and Specificity.The table defines SGP, GTI, and FO-CT+Gaze representations used by compared methods.
- Backbone ablation: DINOv2 achieved the best backbone trade-off, combining 0.9381±0.0261 inter-patient separation with 0.6814 representational variance.Med3D showed representational collapse, UniMISS weak inter-patient separation, and SwinUNETR near-saturated consecutive similarity.
- Ablation analysis: The ablation evaluated attention modes and pooling strategies after an MLP with frozen DINOv2 embeddings produced near-random prediction.The frozen-embedding MLP achieved Mean Val AUC 0.4455 ± 0.045 and test AUC 0.5234.
5 Discussion and Conclusion
The proposed model achieved the strongest reported overall performance, while ablations indicate that gaze integration at both attention and pooling levels is complementary. The authors frame the approach as supporting objective expertise assessment, while noting that broader validation requires more participants.
- Results: 0.91 ROC-AUC, 0.86 F1, and 0.93 specificity were the proposed model’s highest reported values across almost all metrics.An adapted TF-CNN configuration reached the highest sensitivity, 0.92, but had lower specificity of 0.61.
- Comparison with prior methods: CT-Searcher achieved ROC-AUC 0.88 despite being designed for scanpath prediction rather than explicit skill classification.The comparison indicates that scanpath-modeling representations can encode skill-relevant information.
- Ablation analysis: Gaze-Bias plus Gaze-Weighted pooling achieved AUC = 0.91, outperforming Gaze-Bias plus CLS at 0.89 and None plus Gaze-Weighted at 0.86.The ablation supports complementary contributions from gaze-guided attention and gaze-weighted aggregation.
- Discussion and conclusion: The reported clinical significance is objective image-reading assessment, with potential use for gaze-based trainee feedback and perceptually tailored computer-aided detection.These implications are presented within the scope of identifying visual weaknesses and perceptual gaps in less experienced readers.
- Limitations and future work: Future work must increase the participant count beyond five radiologists to support more extensive clinical validation.This is the authors’ stated future direction and scope boundary.