Source-linked AI summary
A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation
Yihua Cheng, Shiyao Huang, Fei Wang, Chen Qian, Feng Lu
TL;DR
Prior gaze-estimation methods overlooked the intrinsic relationship between face and eye features despite using them together. The paper proposes CA-Net, which estimates a face-based basic direction and refines it with eye-based residuals using a bi-gram model and attention. CA-Net achieves state-of-the-art performance on MPIIGaze and EyeDiap, including 4.1° on MPIIGaze.
Problem
Methods combining face and eye images treat them as independent or parallel feature sources, overlooking their intrinsic feature relationship.
Method
CA-Net estimates a basic gaze direction from face features, refines it with eye-image residuals, and uses a bi-gram model plus attention to couple and select these features.
Results
CA-Net achieves state-of-the-art performance on MPIIGaze and EyeDiap, reaching 4.1° on MPIIGaze.
Takeaways & Limitations
The paper provides a coarse-to-fine framework that explicitly considers the correlation between face and eye images for 3D gaze estimation.
Abstract
from arXiv · showhide
Human gaze is essential for various appealing applications. Aiming at more accurate gaze estimation, a series of recent works propose to utilize face and eye images simultaneously. Nevertheless, face and eye images only serve as independent or parallel feature sources in those works, the intrinsic correlation between their features is overlooked. In this paper we make the following contributions: 1) We propose a coarse-to-fine strategy which estimates a basic gaze direction from face image and refines it with corresponding residual predicted from eye images. 2) Guided by the proposed strategy, we design a framework which introduces a bi-gram model to bridge gaze residual and basic gaze direction, and an attention component to adaptively acquire suitable fine-grained feature. 3) Integrating the above innovations, we construct a coarse-to-fine adaptive network named CA-Net and achieve state-of-the-art performances on MPIIGaze and EyeDiap.
Introduction
The paper addresses the overlooked relationship between face and eye features by proposing a coarse-to-fine gaze estimation strategy and the CA-Net framework. Face images provide a basic gaze direction, while eye images provide residual refinement through a bi-gram model and attention component.
- Motivation: Previous methods using face and eye images treat them as independent or parallel feature sources, overlooking their intrinsic relationship at feature granularity.Eye images provide fine-grained gaze-focused features, whereas face images provide coarse-grained features with richer information.
- Coarse-to-fine strategy: The coarse-to-fine strategy estimates a basic gaze direction from the face image and refines it with a residual predicted from eye images.The final gaze direction is obtained by combining the basic gaze direction and gaze residual vectorially.
- Framework: The framework uses a bi-gram model to couple gaze residuals with their corresponding basic gaze directions.This design addresses the problem of ensuring that each residual effectively refines its associated base direction.
- Framework: An attention component adaptively acquires fine-grained eye features suitable for estimating gaze residuals.The component addresses the challenge of selecting effective eye-derived features for residual prediction.
- CA-Net: CA-Net integrates the strategy, bi-gram model, and attention component for coarse-to-fine 3D gaze estimation.The paper reports state-of-the-art performance on the MPIIGaze and EyeDiap benchmarks.
Related work
Appearance-based gaze estimation learns gaze from images and is attractive because it can operate with loose capture requirements. CNN-based methods improved accuracy, progressing from eye-only inputs toward face and eye inputs, but the feature relationship remained insufficiently addressed.
- Method categories: Gaze estimation methods are divided into model-based and appearance-based approaches.Model-based methods build geometric eye models, while appearance-based methods learn mappings from images to gaze.
- Model-based methods: Model-based methods can be accurate but may require dedicated devices such as infrared lights, stereo or high-definition cameras, and RGB-D cameras.Their eye-feature detection requirements constrain deployment settings.
- Appearance-based methods: Appearance-based methods generally require only a webcam and learn a mapping from captured images to corresponding gaze.This looser hardware requirement has attracted substantial research attention.
- CNN-based methods: CNN-based methods improved accuracy over conventional appearance-based methods, initially estimating gaze from eye images.Later work incorporated face images, including approaches using face and eye images together.
- Remaining gap: Recent methods combine face and eye images, but prior work treated their features as independent or parallel sources rather than modeling their intrinsic feature relationship.This limitation motivates the paper’s coarse-to-fine formulation.
Method
CA-Net estimates gaze in a coarse-to-fine manner by combining a face-derived basic gaze direction with eye-derived residuals. Its attention component selects suitable two-eye features, while a bi-gram model couples residual estimation with the basic direction.
- CA-Net: The final CA-Net output is formed by combining the basic gaze direction and gaze residual.The architecture is designed to estimate 3D gaze directions in a coarse-to-fine way.
- Feature generation: The attention component adaptively assigns weights to left and right eye features before fusing them into a suitable eye feature.The weights are produced from scores based on eye features and face-derived information, then balanced with softmax.
- Coarse-to-fine gaze estimation: The bi-gram model bridges basic gaze directions and gaze residuals by passing the face-derived state into the eye-feature processing stage.The first state estimates the basic direction, while the next state uses that state with eye features to estimate residuals.
- Coarse-to-fine gaze estimation: The method uses learned gate functions to filter previous states while incorporating the corresponding feature at each coarse-to-fine stage.The gate function is implemented with GRU in this work.
- CA-Net: CA-Net contains Face-Net and Eye-Net, which estimate the basic gaze direction and gaze residual from face and eye images, respectively.The final gaze direction combines these two estimates.
Experiment
Experiments evaluate CA-Net on MPIIGaze and EyeDiap using leave-one-person-out comparisons and ablations. CA-Net achieves the best reported performance in both datasets, while component and strategy ablations support the roles of attention, gating, and coarse-to-fine refinement.
- Comparison with appearance based methods: 4.1°: CA-Net achieves state-of-the-art performance on MPIIGaze, improving over RT-Gene and RT-Gene (4 ensemble) without ensemble inference.The reported improvements are 0.7° over RT-Gene and 0.2° over RT-Gene (4 ensemble).
- Comparison with appearance based methods: 5.3°: CA-Net achieves the best performance on EyeDiap, improving by 0.6° over Dilated-Net.Dilated-Net matches RT-Gene (4 ensemble) at 5.9°, while Spatial weights CNN and RT-Gene report 6.6° and 6.4°.
- Ablation study: 0.32° and 0.46° decreases occur when removing the learned gate and attention component, respectively, from CA-Net.The ablations evaluate direct face-eye feature concatenation and fixed equal eye-feature weights.
- Ablation study: 4.14°: CA-Net outperforms Face-Net (CA) by 0.51° and Face-Net by 0.44° in the network ablation study.Face-Net reports 4.58°, while Face-Net (CA) reports 4.65°; the comparison tests the coarse-to-fine strategy.
- Ablation study: 0.86°: CA-Net improves over Joint-Net despite using the same backbone, supporting the contribution of the coarse-to-fine strategy.The comparison is reported in the MPIIGaze ablation analysis.
- Additional analysis: 0.49°: Fine-to-coarse performs worse than CA-Net, while One gram is 0.29° worse than CA-Net in additional MPIIGaze analyses.Fine-to-coarse estimates the basic gaze direction from eye images and refines it with face-image residuals; One gram lacks basic gaze-direction information for refinement.
Conclusion
The paper proposes CA-Net, combining a coarse-to-fine gaze strategy with attention and a bi-gram model, and reports state-of-the-art performance on MPIIGaze and EyeDiap.
- CA-Net estimates 3D gaze directions by refining a face-derived basic gaze direction with an eye-derived residual.The coarse-to-fine process uses face images for approximation and eye images for refinement.
- An attention component adaptively assigns weights to eye images to obtain suitable fine-grained features for gaze-residual estimation.
- A bi-gram model bridges basic gaze directions and gaze residuals within the coarse-to-fine process.
- CA-Net achieves state-of-the-art performance on MPIIGaze and EyeDiap.