Source-linked AI summary
PupilNet: Convolutional Neural Networks for Robust Pupil Detection
Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, Enkelejda Kasneci
TL;DR
Natural-scene artifacts make reliable pupil detection difficult, although robust detection is needed for pervasive video-based eye tracking. The paper proposes a dual-CNN pipeline that first estimates a coarse location from downscaled image patches and then refines it in the original image. It reports substantial improvement over state-of-the-art approaches on challenging real-world recordings, while noting a training trade-off between batch and stochastic learning.
Problem
Natural-environment artifacts can lower pupil-detection rates, making real-time, accurate, and robust detection an essential prerequisite for pervasive video-based eye tracking.
Source: §1 · pp. 1–2Method
A dual-CNN pipeline estimates a coarse pupil location from downscaled-image subregions, then refines it using subregions around that estimate in the original image.
Source: §1 · p. 2Results
The CNN pipeline outperformed state-of-the-art approaches by a large margin on real-world recordings, evaluated using over 79,000 hand-labeled images.
Source: §6 · p. 8Takeaways & Limitations
The authors report that the approach avoids high computational costs and is runnable in real time on hardware without an accessible GPU.
Source: §6 · p. 8Stochastic gradient descent can make damaging updates when training examples are inadequate, so the authors favor batch learning to dilute this error.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Real-time, accurate, and robust pupil detection is an essential prerequisite for pervasive video-based eye-tracking. However, automated pupil detection in real-world scenarios has proven to be an intricate challenge due to fast illumination changes, pupil occlusion, non centered and off-axis eye recording, and physiological eye characteristics. In this paper, we propose and evaluate a method based on a novel dual convolutional neural network pipeline. In its first stage the pipeline performs coarse pupil position identification using a convolutional neural network and subregions from a downscaled input image to decrease computational costs. Using subregions derived from a small window around the initial pupil position estimate, the second pipeline stage employs another convolutional neural network to refine this position, resulting in an increased pupil detection rate up to 25% in comparison with the best performing state-of-the-art algorithm. Annotated data sets can be made available upon request.
1. Introduction
Video-based eye tracking struggles in natural environments, where image artifacts can lower pupil-detection rates and require laborious manual post-processing. The paper proposes a dual-CNN approach and online training-data generation to improve robust, efficient pupil detection.
1. Introduction
- Natural-scene artifacts—including changing illumination, motion blur, reflections, off-axis views, and physiological features—make pupil detection unreliable and can necessitate laborious manual post-processing.Manual post-processing is impossible for real-time applications that rely on pupil monitoring.Source: §1 · p. 1
- Real-time, accurate, and robust pupil detection is an essential prerequisite for pervasive video-based eye tracking.Source: §1 · p. 2
- Current algorithms can have detection rates as low as 34% in realistic use cases, while the authors report considerable improvements from their CNN approach.Source: §1 · p. 2
- The proposed dual-CNN pipeline estimates a coarse pupil location from downscaled-image subregions, then refines it using subregions around that estimate in the original image.The coarse estimate narrows the second stage’s search region, mitigating noise and reducing computational costs.Source: §1 · p. 2
- The authors propose online generation of training data for pupil-center detection in online scenarios, and report that the method can run in real time without an accessible GPU.Source: §1 · p. 2
2. Related Work
Prior pupil-detection methods combine thresholding, geometric fitting, and artifact-specific processing. Although newer methods address natural-environment challenges, robust detection remains difficult because noise persists.
2. Related Work
- Earlier methods use thresholding and mass centers, feature- or model-based ellipse fitting, or coarse positioning followed by pupil-center estimation.These approaches showed good detection rates and robustness in controlled laboratory settings.Source: §2 · p. 2
- SET [10] and ExCuSe explicitly target natural-environment challenges using, respectively, luminance segmentation with ellipse selection and reflection analysis with contour extraction.Source: §2 · pp. 2–3
- Despite reported improvements over earlier methods, noise remains a major issue and robust detection for online real-world applications remains an open challenge.Source: §2 · p. 3
3. Method
The method localizes the pupil in two stages: a CNN searches downscaled-image subregions for a coarse position, and a second CNN refines it in the original image. The training procedures use labeled subregions from existing and additional datasets.
3. Method
- The coarse CNN scans overlapping patches from a downscaled image, and the second CNN selects the highest-rated patch around that estimate in the original image.Downscaling reduces noise and computational cost; the second stage improves precision lost through downscaling.Source: §3 · p. 3, Figure 2 · p. 3
3.1. Coarse Positioning Stage
- The first stage downsamples 384×288-pixel input images fourfold and rates 24×24-pixel patches shifted one pixel at a time to estimate the coarse pupil location.The center of the highest-rated patch is selected; ratings represent confidence that the pupil center lies within the patch.Source: §3.1 · p. 3, Figure 3 · p. 3
- The coarse CNN uses convolution, average pooling, a fully connected layer, and a single perceptron; pooling makes learned features more robust to small translations and blur.Source: §3.1 · pp. 3–4, Figure 4 · p. 4
3.2. Fine Positioning Stage
- The second CNN evaluates 89×89-pixel patches around the coarse estimate, selecting the highest-rated center to refine the pupil location despite the coarser stage’s localization error.The authors report that the higher-cost second stage remains efficient on conventional mobile computers.Source: §3.2 · p. 4
3.3. CNN Training Methodology
- Batch gradient descent was chosen for adaptive training because stochastic gradient descent can make damaging updates when examples are inadequate, while batch training dilutes that error.Source: §3.3 · p. 4
3.3.1 Coarse Positioning CNN
- Coarse-CNN training labels patches centered within one pixel of the hand-labeled pupil center as valid and those farther away, up to five pixels, as invalid.The procedure yields nine valid and 32 invalid samples per labeled image; training sets vary whether new datasets are included.Source: §3.3.1 · pp. 4–5, Figure 5 · p. 5
3.3.2 Fine Positioning CNN
- Fine-CNN training uses one valid patch at the labeled center and eight invalid patches centered five pixels away, with a subset of generated samples selected for training.The reduced sample count constrains learning time, memory, and storage consumption.Source: §3.3.2 · p. 5
4. Data Set
The evaluation dataset combines existing data with five additional hand-labeled datasets collected in challenging real-world conditions.
4. Data Set
- The study uses the datasets introduced by Fuhl et al. and supplements them with five additional hand-labeled datasets.Source: §4 · p. 5
- The additional datasets include 41,217 images collected during public-road driving sessions and contain changing illumination, spectacle reflections, and disruptive physiological features.Source: §4 · p. 5
- Dataset examples distinguish a small fraction of non-challenging, laboratory-like samples from challenging samples with natural-environment artifacts.Each figure column represents a distinct dataset.Source: Figure 6 · p. 5
5. Evaluation
The evaluation compares coarse and fine CNN configurations, then examines learned filters and weights. FK8P8 and SK8P8 outperform the best-performing state-of-the-art algorithm across the evaluated cases.
5. Evaluation
- Performance was reported against pixel distance from expert-labeled pupil centers, with results discussed at five-pixel error because label imprecision could not be excluded.The evaluation used the average pupil detection rate as a function of that distance.Source: §5 · p. 5
5.1. Coarse Positioning
- More convolutional filters and extended learning affected coarse-CNN performance more than additional fully connected perceptrons, though filters remained unsaturated and computationally costly.The comparison used CK4P8, CK8P8, and CK16P32 for filter counts, and CK8P16 versus CK8P16ext for extended learning.Source: §5.1 · p. 6, Figure 7 · p. 6
5.2. Fine Positioning
- FK8P8 and SK8P8 surpassed the best-performing state-of-the-art algorithm by approximately 25% and 15%, respectively, across evaluation cases.The cases covered Fuhl et al. data, all data sets, and images not used for training.Source: §5.2 · p. 7, Figure 9 · p. 8
- Coarse approaches mostly improved on state-of-the-art algorithms, but CK8P8ray showed no significant gain over coarse positioning.The authors attribute the hybrid method's result to traditional detection methods remaining affected by the stated challenges.Source: §5.2 · p. 7
5.3. CNN Operational Behavior Analysis
- CK8P8 learned rounded, edge-like filters and partial inverse pairs; its weight maps suggest filters g and h may respond to off-center pupils, while center-surround filter e may ignore small blobs.The figures show filter weights and responses, plus the fully connected layer's weight maps.Source: §5.3 · pp. 7–8, Figure 10 · p. 7, Figure 11 · p. 8
6. Conclusion
The paper reports that its CNN pipeline outperforms state-of-the-art approaches at low computational cost, based on a large set of hand-labeled real-world recordings. Future work will investigate online use, where continuous parameter adaptation remains a challenge.
6. Conclusion
- The proposed CNN pipeline outperformed state-of-the-art approaches by a large margin while avoiding high computational costs.The evaluation used more than 79,000 hand-labeled images from real-world recordings with reflections, changing illumination, and occlusion.Source: §6 · p. 8
- The evaluation used over 79,000 hand-labeled images, including 41,000 that were complementary to images from the literature.The recordings included real-world artifacts such as reflections, changing illumination, and occlusion.Source: §6 · p. 8
- Future work will investigate applying the pipeline online, where continuous parameter adaptation is an additional challenge.Source: §6 · p. 8