Source-linked AI summary
GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
Mohammed Oussama Benyahia, Marouane Tliba, Mohamed Amine Kerkouri, Taifour Yousra, Bin Wang, Max Bengtsson, Gorkem Durak, Elif Keles, Zuheng Ming, Marek Penhaker, Azeddine Beghdadi, Ulas Bagci, Aladine Chetouani
TL;DR
Medical image segmentation remains difficult to scale because strong methods commonly require dense expert annotations and task-specific training. GazeRefine instead uses clinician gaze to initialize and refine prototypes in frozen DINOv3 features for zero-shot segmentation, achieving strong colonoscopy performance and competitive prostate MRI performance.
Problem
High-performing medical image segmentation methods still depend on costly dense annotations and task-specific training, making scalable deployment across clinical settings difficult.
Method
GazeRefine converts duration-weighted clinician fixations into foreground and background priors, then refines frozen DINOv3 prototypes using discrimination, affinity propagation, and gaze anchoring.
Results
88.10% ± 0.14 Dice on Kvasir-SEG, approaching nnU-Net at 88.41%±0.47 and exceeding GazeMedSAMv2 at 85.19% ± 0.13; NCI-ISBI Dice was 80.28% ± 0.11.
Takeaways & Limitations
The results support clinician gaze as a way to steer frozen visual representations for label-efficient, human-in-the-loop medical image segmentation.
Abstract
from arXiv · showhide
Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git
1 Introduction
GazeRefine addresses the annotation and task-specific training burden in medical image segmentation by using clinician gaze to steer frozen self-supervised features for zero-shot segmentation. Its iterative prototype refinement extends predictions beyond fixated regions while avoiding segmentation-specific training components.
- 1 Introduction: Dense expert masks and task-specific training remain central dependencies of high-performing medical segmentation methods, limiting scalability across settings.Annotations are costly, time-consuming, variable across observers, and difficult to scale across organs, modalities, institutions, and rare pathologies.
- 1 Introduction: Weak supervision reduces mask requirements, but typically still requires task-specific training or optimization, while promptable models use segmentation-specific architectures.Self-supervised vision transformers instead provide dense patch representations with emergent localization.
- 1 Introduction: Clinician gaze captures task-relevant visual attention and can indicate suspicious or clinically meaningful regions during image interpretation.Prior methods use gaze either as weak supervision for training or as an input prompt to SAM-based models.
- 1 Introduction: GazeRefine uses sparse clinician gaze as an inference-time prompt for zero-shot segmentation in frozen DINOv3 feature space.Duration-weighted fixations form foreground and background priors that initialize semantic prototypes.
- 1 Introduction: The framework is training-free and segmentation-label-free, requiring no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates.It is evaluated on gaze-annotated polyp segmentation and prostate MRI segmentation benchmarks.
- 1 Introduction: Foreground–background discrimination and kNN affinity propagation repeatedly refine gaze-derived prototypes, enabling segmentation beyond directly fixated regions.The framework combines these refinements with anchoring to the original gaze guidance.
2 Proposed Method
GazeRefine uses clinician gaze and frozen DINOv3 features to initialize foreground and background prototypes, then iteratively refines them into a dense segmentation mask. The refinement combines foreground–background discrimination, kNN affinity propagation, and gaze anchoring to extend predictions beyond fixated regions while limiting drift.
- Feature Representation and Clinician Gaze Modeling: The pipeline extracts normalized DINOv3 patch embeddings and projects duration-weighted clinician gaze onto the patch grid as a spatial prior.The resulting gaze map is normalized to [0, 1], producing patch weights h_i.
- Gaze-Guided Prototype Initialization: Foreground and complementary background gaze weights initialize semantic prototypes through weighted pooling in frozen DINOv3 feature space.These prototypes serve as gaze-derived foreground and background anchors.
- Gaze-Anchored Prototype Refinement: At each iteration, patch similarities to foreground and background prototypes produce non-negative foreground confidence scores for discrimination.Because embeddings and prototypes are normalized, the inner products represent cosine similarities.
- Gaze-Anchored Prototype Refinement: kNN affinity propagation refines confidence scores across neighboring patches, while updated foreground and background prototypes are recomputed from the refined weights.The affinity distribution among neighboring patches is controlled by τ.
- Gaze-Anchored Prototype Refinement: Current prototypes are blended with initial gaze-derived anchors to limit semantic drift, and refinement stops after T iterations or prototype convergence.Convergence requires both foreground and background prototype changes to fall below ε.
- Final Segmentation and Training-Free Inference: The final score map is reshaped, upsampled to image resolution, normalized, and thresholded at θ = 0.5 to produce the binary mask.The four-stage framework proceeds from feature extraction and gaze modeling through prototype initialization, recurrent refinement, and final mask generation.
3 Experimental Setup
GazeRefine is evaluated on gaze-annotated polyp and prostate segmentation datasets using established benchmark splits. The same core refinement hyperparameters are used across datasets, with dataset-specific Gaussian gaze-map spreads.
- Datasets: The evaluation uses 1,000 Kvasir-SEG colonoscopy images and 789 NCI-ISBI prostate MRI slices with gaze annotations.After filtering short and out-of-bounds fixations, images contain 5–99 and slices contain 6–54 fixations, respectively.
- Implementation Details: The method uses λ = 0.5, T = 5, k = 5, τ = 0.1, ε = 10−6, and θ = 0.5 for both datasets.The Gaussian spread is σ = 2.0 for Kvasir-SEG and σ = 0.5 for NCI-ISBI.
- Evaluation Protocol: Established trainable-baseline splits are 90%/10% for Kvasir-SEG and 87%/13% for NCI-ISBI.GazeRefine is evaluated using the benchmark protocols described for each dataset.
4 Experimental Results
GazeRefine is evaluated against supervised, weakly supervised, gaze-supervised, and zero-shot gaze-guided methods on Kvasir-SEG and NCI-ISBI, with strong polyp-segmentation results and lower prostate-MRI performance. Ablations show that background cleaning is the dominant refinement component, while kNN propagation provides complementary gains.
- GazeRefine vs. State-of-the-Art Methods: 88.10% ± 0.14 Dice on Kvasir-SEG closely approaches nnU-Net’s 88.41%±0.47 while exceeding GazeMedSAMv2’s 85.19% ± 0.13.The nnU-Net comparison uses 90% of annotated data for training, whereas GazeRefine is zero-shot.
- GazeRefine vs. State-of-the-Art Methods: Qualitative comparisons on representative Kvasir-SEG and NCI-ISBI samples assess GazeRefine against gaze-guided zero-shot segmentation methods.
- GazeRefine vs. State-of-the-Art Methods: 80.28% ± 0.11 mean Dice on NCI-ISBI is below the best zero-shot gaze-guided result of 85.62% ± 0.10 but remains competitive with several weakly supervised baselines.The authors associate the lower prostate-MRI performance with limited separability of low-contrast anatomical boundaries in general-purpose feature spaces.
- Ablation Study: Removing kNN propagation reduces Dice from 88.10% to 87.85% on Kvasir-SEG and from 80.28% to 73.55% on NCI-ISBI.The larger prostate-MRI decrease indicates that affinity propagation is particularly beneficial there.
- Ablation Study: Disabling background cleaning causes a much larger performance drop on both datasets, identifying it as the dominant component for limiting background contamination.Removing both components remains close to the setting without background cleaning, while kNN propagation supplies complementary gains.
5 Discussion and Conclusion
GazeRefine performs segmentation from frozen DINOv3 representations using clinician gaze to initialize and refine foreground and background prototypes. It achieves strong zero-shot performance on Kvasir-SEG, while NCI-ISBI exposes challenges for low-contrast prostate MRI.
- The method avoids SAM-like models, prompt encoders, fine-tuning, and gradient updates while converting sparse gaze into coherent masks.
- Strong Kvasir-SEG performance contrasts with lower NCI-ISBI performance, where general-purpose features face low-contrast prostate-MRI boundaries.
- The ablation study identifies background cleaning as critical for limiting prototype contamination and kNN propagation as improving spatial coherence.