Source-linked AI summary
ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments
Wolfgang Fuhl, Thiago C. Santini, Thomas Kuebler, Enkelejda Kasneci
TL;DR
Real-world pupil detection is challenged by illumination changes, reflections, and recording conditions that limit laboratory-oriented approaches. ElSe evaluates ellipses in filtered edge images, with coarse positioning as a fallback, and was reported to outperform four state-of-the-art methods on challenging hand-labeled eye images.
Problem
Pupil detection for real-world eye tracking is limited by changing illumination, reflections, motion blur, eyelashes, and off-axial recording conditions.
Method
ElSe filters grayscale eye images, evaluates connected edges and fitted ellipses, and applies convolution-based coarse positioning when edge detection fails.
Results
ElSe outperformed four state-of-the-art methods on 94,713 challenging, hand-labeled eye images.
Takeaways & Limitations
ElSe is presented as a real-time pupil detection algorithm for outdoor eye-tracking experiments, with implementation and evaluation datasets available for download.
Takeaways & Limitations
ElSe fails when reflections substantially cover the pupil or when image intensity patterns violate its edge and coarse-positioning assumptions.
Abstract
from arXiv · showhide
Fast and robust pupil detection is an essential prerequisite for video-based eye-tracking in real-world settings. Several algorithms for image-based pupil detection have been proposed, their applicability is mostly limited to laboratory conditions. In realworld scenarios, automated pupil detection has to face various challenges, such as illumination changes, reflections (on glasses), make-up, non-centered eye recording, and physiological eye characteristics. We propose ElSe, a novel algorithm based on ellipse evaluation of a filtered edge image. We aim at a robust, resource-saving approach that can be integrated in embedded architectures e.g. driving. The proposed algorithm was evaluated against four state-of-the-art methods on over 93,000 hand-labeled images from which 55,000 are new images contributed by this work. On average, the proposed method achieved a 14.53% improvement on the detection rate relative to the best state-of-the-art performer. download:ftp://emmapupildata@messor.informatik.unituebingen. de (password:eyedata).
1 Introduction
Mobile eye trackers support gaze measurement in real-world tasks, but pupil detection becomes difficult outside laboratory conditions because of environmental and recording-related noise.
- 1 Introduction: Real-world eye tracking encounters difficulties that are less prominent under laboratory conditions.Reported applications include driving and shopping.
- 1 Introduction: Changing illumination, motion blur, recording errors, and eyelashes covering the pupil are reported sources of pupil-detection difficulty.Rapid illumination changes can occur when subjects move quickly or rotate relative to uneven light sources.
- 1 Introduction: Glasses or contact lenses can introduce reflections, while off-axial eye cameras create an additional recording challenge.These conditions contribute to low pupil-detection rates outside the laboratory.
- 1 Introduction: Non-robust pupil signals can make collected eye-tracking data require manual post-processing, which is laborious and unsuitable for real-time monitoring.Real-time applications also constrain algorithms because computer-intensive methods may be impracticable.
2 Method
ElSe processes grayscale eye images by filtering and refining edges, then evaluates candidate contours as ellipses before using a secondary analysis when no valid ellipse is found.
- 2 Method: ElSe excludes 10% of the image border, normalizes the grayscale image, and applies a Canny edge filter.The border exclusion reduces the effect of eyeglass frames.
- 2 Method: The algorithm removes interfering edge connections and evaluates connected edges using geometric, intensity, ellipse-fitting, and pupil-plausibility criteria.A valid ellipse describing the pupil is returned when one is found.
- 2 Method: When no suitable ellipse is found, ElSe conducts a second analysis based on coarse positioning and convolution.This fallback addresses cases where edge filtering does not produce suitable edges.
2.1 Filter edges
ElSe refines the edge image by thinning and straightening lines, removing problematic connections, and splitting contours where their course is inconsistent with a common ellipse.
- 2.1 Filter edges: The morphologic approach thins and straightens edges, then removes pixels involved in orthogonal connections or having more than two neighbors.These operations improve the breaking procedure for candidate contours.
- 2.1 Filter edges: Both morphologic and algorithmic edge-filtering approaches produce comparable results.ElSe uses the morphologic approach in its provided implementation because it requires less computational power.
- 2.1 Filter edges: Morphologic patterns remove or add edge pixels to thin lines, straighten them, and break up orthogonal connections.The implementation uses this approach because it requires less computational power.
- 2.1 Filter edges: The algorithmic approach splits lines where their course cannot belong to a common ellipse after edge thinning, neighbor filtering, and line straightening.It compares the evolving angle and distance between an orthogonal vector and a vector from the line’s starting point.
2.2 Select best ellipse
ElSe selects pupil candidates by collecting edge lines, rejecting straight or implausible contours, fitting ellipses, and ranking the remaining ellipses by intensity and shape.
- 2.2 Select best ellipse: Consecutive edge pixels are collected into lines before candidate contours are evaluated.The morphologic approach combines connected edge pixels into a line, while the algorithmic approach can separate open and closed lines.
- 2.2 Select best ellipse: Straight lines are rejected because pupil contours are expected to be round or elliptical.Straightness is assessed from the mean position of the line’s pixels using min_mean_line_dist in both dimensions.
- 2.2 Select best ellipse: Ellipse evaluation compares radius shape, pupil-area plausibility, and the intensity difference between inner and surrounding regions.The implementation uses a validitythreshold of 10 for the surface difference.
- 2.2 Select best ellipse: Candidate lines are fitted with least squares ellipses, chosen for efficient fitting in low-latency online scenarios.The resulting ellipse is then subjected to further validity checks.
- 2.2 Select best ellipse: The selected ellipse has the lowest inner gray value and the roundest shape, with the clearest contour breaking ties.The ranking formula uses ellipse radii and the calculated inner gray value.
2.3 Coarse positioning
ElSe performs coarse pupil positioning by downscaling the image with intensity-dependent filtering, then combining circular mean and surface-difference convolution responses.
- 2.3 Coarse positioning: A fallback coarse-positioning method rescales the image before convolution to keep runtime tractable when reliable pupil edges are unavailable.It is used for cases such as motion blur, dark surroundings, or eyelashes obscuring the pupil edge.
- 2.3 Coarse positioning: Downscaling uses the mean of the lower-intensity histogram region within each influencing neighborhood, thereby weighting dark pixels more strongly.The lower region is defined relative to the mean intensity of the whole neighborhood.
- 2.3 Coarse positioning: The implementation uses overlapping neighborhoods with radiusscale = 5, which downscales the image by a factor of six.The overlapping regions exclude the centers of neighboring boxes.
- 2.3 Coarse positioning: The coarse-positioning filters use circular mean and surface-difference kernels sized from image resolution and expected pupil dimensions.The surface-difference filter is designed to respond to dark centers surrounded by brighter regions.
- 2.3 Coarse positioning: The two convolution results are combined by point-wise multiplication, and the absolute maximum identifies the coarse pupil position.The workflow visualizes this maximum as a white cross in the original image.
2.4 Optimize position
The optimized pupil position is obtained within a restricted image region by thresholding pixels and selecting the resulting candidate position.
- 2.4 Optimize position: Optimization is confined to the area enclosed by a white box, where pixels below the calculated threshold determine the resulting position.The selected position is marked by a white cross.
2.5 Validate position
ElSe validates the coarse candidate because the fallback method always produces a pupil location, including when the eye is closed.
- 2.5 Validate position: The candidate location is validated using the ellipse-validation procedure, with ellipse diameters set to radiusfilter ∗radiusfilter ∗2 + 1.The validitythreshold is set to 10.
3 Experimental Evaluation
ElSe was evaluated on more than 94,000 hand-labeled images spanning prior and newly collected real-world eye-tracking data, and compared with four state-of-the-art methods. It outperformed competitors overall while retaining identifiable failure cases and parameter-dependent runtime trade-offs.
- 3 Experimental Evaluation: More than 94,000 hand-labeled eye images were used, including related-work data and new sets from driving and indoor experiments.The new data include challenges from eyelids, eyelashes, glasses reflections, and other real-world conditions.
- 3 Experimental Evaluation: ElSe was compared with SET, Starburst, Swirski, and ExCuSe using fixed parameters across all data sets.SET was evaluated with two parameter combinations, while the other implementations used their provided settings.
- 3 Experimental Evaluation: ElSe clearly outperformed all competitor algorithms in both image-weighted and equally weighted data-set evaluations across pixel-error thresholds.It was not best on six listed data sets but remained on average 2% worse than the best performer in those cases.
- 3 Experimental Evaluation: Changing radiusscale by too much or too little reduced detection rates by about 1-2%, while also changing convolution runtime.Higher values reduced runtime and lower values increased it, yet ElSe remained above its competitors in the reported comparison.
- 3 Experimental Evaluation: ElSe failed when reflections damaged the edge response or altered the convolution response, including partial reflections and reflections covering most of the pupil.The failure cases are illustrated through input images, filtered edge images, and convolution responses.
4 Conclusions
ElSe is presented as a real-time pupil detection algorithm evaluated on challenging outdoor eye images with noise sources including reflections, changing illumination, and off-axial camera position.
- 94,713 challenging, hand-labeled eye images were used to evaluate ElSe under reflections, changing illumination, off-axial camera position, and other noise sources.
- ElSe outperformed four state-of-the-art methods by far in the reported comparison.
- The implementation and evaluation datasets are available for download to support application in outdoor eye-tracking experiments.