Source-linked AI summary
Appearance-Based Gaze Estimation Using Dilated-Convolutions
Zhaokang Chen, Bertram E. Shi
TL;DR
Appearance-based gaze estimation must resolve subtle eye-image changes while existing CNN architectures reduce high-level spatial resolution through downsampling. The paper uses dilated convolutions to extract high-resolution features and evaluates Dilated-Nets cross-subject on Columbia Gaze and MPIIGaze, achieving significant gains and state-of-the-art results on both datasets.
Problem
Current gaze-estimation architectures may lose subtle gaze information because pooling and other downsampling reduce high-level feature-map resolution.
Method
Dilated-Net uses dilated-convolutional eye networks to extract high-level features at high spatial resolution, with a multi-region variant combining face and both-eye inputs.
Results
Dilated-Nets significantly outperform similar non-dilated CNNs by 3.2% to 20.8% depending on the task and achieve state-of-the-art results on both datasets.
Takeaways & Limitations
High-resolution features extracted with dilated convolutions benefit appearance-based gaze estimation in cross-subject experiments on Columbia Gaze and MPIIGaze.
Abstract
from arXiv · showhide
Appearance-based gaze estimation has attracted more and more attention because of its wide range of applications. The use of deep convolutional neural networks has improved the accuracy significantly. In order to improve the estimation accuracy further, we focus on extracting better features from eye images. Relatively large changes in gaze angles may result in relatively small changes in eye appearance. We argue that current architectures for gaze estimation may not be able to capture such small changes, as they apply multiple pooling layers or other downsampling layers so that the spatial resolution of the high-level layers is reduced significantly. To evaluate whether the use of features extracted at high resolution can benefit gaze estimation, we adopt dilated-convolutions to extract high-level features without reducing spatial resolution. In cross-subject experiments on the Columbia Gaze dataset for eye contact detection and the MPIIGaze dataset for 3D gaze vector regression, the resulting Dilated-Nets achieve significant (up to 20.8%) gains when compared to similar networks without dilated-convolutions. Our proposed Dilated-Net achieves state-of-the-art results on both the Columbia Gaze and the MPIIGaze datasets.
1 Introduction
Appearance-based gaze estimation uses RGB images for flexible, unconstrained tracking but remains difficult because appearance, head-eye relationships, gaze ranges, and illumination vary. The paper proposes high-resolution eye features with dilated convolutions to capture subtle gaze-related changes and reports gains across two datasets.
- Unconstrained gaze tracking aims to operate independently of calibration, subject, viewpoint, and illumination.
- Appearance-based gaze estimation predicts 2D gaze targets or 3D gaze angles from RGB images captured by inexpensive, widely available cameras.
- The task remains challenging because individual appearance, head-eye relationships, gaze ranges, and illumination conditions vary.
- Deep CNNs have improved appearance-based gaze tracking using public real and synthetic datasets, but further improvement remains possible.
- The proposed Dilated-Net extracts high-resolution eye features with dilated convolutions, which enlarge receptive fields without reducing spatial resolution.
- Cross-subject experiments on Columbia Gaze and MPIIGaze show gains of 3.2% to 20.8% over similar CNNs and state-of-the-art results on both datasets.
2 Related Work
Appearance-based gaze estimation maps eye or face imagery to gaze targets or directions using increasingly data-driven methods. Related work spans classical learners, deep CNNs, multi-region inputs, gaze classification, and dilated convolutions, while this paper emphasizes better eye features within multi-region networks.
- Appearance-based methods estimate 2D gaze targets or 3D gaze directions from images captured by off-the-shelf cameras.
- Earlier approaches included Random Forests, k-Nearest Neighbors, Support Vector Regression, and deep CNNs for gaze estimation.
- Multi-region CNNs incorporate face, eye, and face-grid inputs, while other networks use full-face features or spatial weighting.
- Gaze tracking has also been formulated as classification, including target-region classification and binary eye-contact detection.
- This paper focuses on improving eye-image features for multi-region networks rather than primarily adding information from other facial regions.
- Dilated convolutions enlarge receptive fields without reducing spatial resolution or increasing parameter count.
3 Methodology
The method preserves high-resolution eye features while expanding receptive fields with dilated convolutions, targeting subtle gaze-related appearance changes that pooling can obscure. These eye features are integrated into a multi-region network alongside face features and evaluated against a matched non-dilated CNN.
- Motivation: Small gaze changes can affect only a few eye-image pixels, motivating high-resolution feature extraction.The paper also notes that eye openness and shape provide gaze information.
- Motivation: Max-pooling progressively reduces feature-map resolution, increasing receptive-field spacing and potentially discarding fine spatial differences.In the compared CNN, receptive-field centers are 8 px apart after repeated pooling.
- Dilated-Convolutions: Dilated convolutions enlarge receptive fields without reducing spatial resolution or substantially increasing parameter count.Dilation rates greater than one expand receptive fields while preserving spatial resolution.
- Dilated-Convolutions: A 3 × 3 dilated convolution with dilation rates (2, 2) produces a 7 × 7 lower-level receptive field while preserving spatial resolution.The lower-level locations contributing to higher-level units remain constant across positions.
- Multi-Region Dilated-Net: The multi-region Dilated-Net processes the face and both eyes through separate networks, using dilated-convolutional layers in the eye networks.Face, left-eye, and right-eye features are concatenated before the final fully connected layers.
- Comparison Network: The non-dilated comparison CNN replaces four dilated-convolutional layers with convolutional layers and three pooling layers while matching final feature-map size and parameter count.The comparison isolates the effect of dilated convolutions.
4 Experiments
Cross-subject experiments evaluated Dilated-Nets for eye-contact detection on Columbia Gaze and 3D gaze regression on MPIIGaze. The models improved over comparable baselines, with gains associated with high-resolution eye features and performance across most subjects.
- Experimental setup: Cross-subject experiments covered eye-contact detection on Columbia Gaze and 3D gaze regression on MPIIGaze.The Columbia evaluation used leave-one-group-out cross-validation over 11 subject groups, while MPIIGaze used leave-one-subject-out cross-validation.
- MPIIGaze: 5.21° on MPIIGaze and 5.12° on MPIIGaze+ were achieved by Dilated-Net (single), outperforming the branched CNN by 0.17° and 0.18°, respectively.Against ImageNet-pretrained CNN (single), the gains were 0.24° on MPIIGaze and 0.23° on MPIIGaze+.
- MPIIGaze: 14.3% lower mean angular error than iTracker (AlexNet) and 11.1% lower error than CNN (multi) were achieved by Dilated-Net (multi) with face-plus-two-eyes input.The comparison used results reported in original space with the face center as origin.
- Feature analysis: Dilated-Net (multi) outperformed CNN (multi) for 12 of 15 subjects, indicating improvement for most subjects despite individual-appearance variation.The feature analysis compared final feature maps and found stronger gaze-angle correlations for Dilated-Net features than for CNN features.
- Efficiency comparison: Dilated-Net (multi) matched the spatial weights CNN while using 96 × 224 inputs, 64% input resolution, and approximately 5 M rather than 196 M parameters.The reported comparison used the same results while highlighting differences in input size, resolution, and parameter count.
5 Conclusion
Dilated-convolutions extract high-level eye-image features at high resolution, helping capture small appearance variations and improving gaze estimation. Cross-subject experiments found significant gains over similar CNNs without dilated-convolutions, with multi-region Dilated-Net achieving state-of-the-art results on both datasets.
- 5 Conclusion: Dilated-convolutions extract high-level features at high resolution, allowing networks to capture small variability in eye images.The approach targets subtle appearance changes associated with gaze direction.
- 5 Conclusion: Cross-subject experiments on the Columbia Gaze and MPIIGaze datasets showed significant gains over similar CNNs without dilated-convolutions.The evaluation covered eye contact detection and gaze estimation across the two datasets.
- 5 Conclusion: The proposed multi-region Dilated-Net achieved state-of-the-art results on both the Columbia Gaze and MPIIGaze datasets.The paper identifies this model as its strongest overall result.
- 5 Conclusion: The authors plan to apply gaze estimation to human-machine and human-robot interaction, using gaze trajectory as a cue for user intent.They describe intent estimation as a basis for more natural system responses and appropriate assistance.