PaperlayerView exact revision on arXiv

Source-linked AI summary

Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark

Yihua Cheng, Haofei Wang, Yiwei Bao, Feng Lu

arXiv:2104.12668v2cs.CV

TL;DR

Appearance-based gaze estimation requires effective feature extraction, robust regression, abundant training data, and solutions for domain shift and calibration. This paper systematically reviews deep-learning methods, organizes them across the gaze-estimation pipeline, and benchmarks approaches while standardizing relevant processing choices.

  • Problem

    Appearance-based gaze estimation must map high-dimensional images to gaze, while collecting training or calibration samples is costly and performance can drop in new domains.

  • Method

    The paper reviews methods across feature extraction, neural-network design, personal calibration, and platforms, and summarizes rectification and gaze-coordinate conversion for fair benchmarking.

  • Results

    The paper provides a comprehensive overview, public-dataset characterization, source code for typical algorithms, and benchmarks comparing state-of-the-art methods.

  • Takeaways & Limitations

    The survey serves as a reference for developing deep-learning gaze-estimation methods and a guideline for future research.

  • Takeaways & Limitations

    Calibration presents a performance–time trade-off: longer calibration can produce more accurate estimates, making fast satisfactory calibration an open direction.

Abstract

from arXiv · show

Human gaze provides valuable information on human focus and intentions, making it a crucial area of research. Recently, deep learning has revolutionized appearance-based gaze estimation. However, due to the unique features of gaze estimation research, such as the unfair comparison between 2D gaze positions and 3D gaze vectors and the different pre-processing and post-processing methods, there is a lack of a definitive guideline for developing deep learning-based gaze estimation algorithms. In this paper, we present a systematic review of the appearance-based gaze estimation methods using deep learning. Firstly, we survey the existing gaze estimation algorithms along the typical gaze estimation pipeline: deep feature extraction, deep learning model design, personal calibration and platforms. Secondly, to fairly compare the performance of different approaches, we summarize the data pre-processing and post-processing methods, including face/eye detection, data rectification, 2D/3D gaze conversion and gaze origin conversion. Finally, we set up a comprehensive benchmark for deep learning-based gaze estimation. We characterize all the public datasets and provide the source code of typical gaze estimation algorithms. This paper serves not only as a reference to develop deep learning-based gaze estimation methods, but also a guideline for future gaze estimation research. The project web page can be found at https://phi-ai.buaa.edu.cn/Gazehub.

1 INTRODUCTION

Gaze estimation supports applications ranging from cognition analysis to XR and human-computer interaction, but appearance-based methods must infer gaze from complex images. This paper systematically reviews deep learning approaches, standardizes processing for fair comparison, and benchmarks state-of-the-art methods.

  • Motivation: Gaze direction is a 3D unit vector from the eye centers toward gaze targets, while point of gaze maps it onto a defined plane.PoG is widely used to indicate attention areas on screens or displays.
  • Motivation: Appearance-based methods use off-the-shelf cameras to capture eye appearance and regress gaze without dedicated geometric-feature devices.Their simple hardware setup requires effective feature extraction, robust regression, and many training samples.
  • Motivation: Deep learning automatically extracts high-level features and models nonlinear mappings from appearance to gaze, including under large head motion.The paper states that these advantages improve accuracy, robustness, and cross-subject performance relative to conventional appearance-based methods.
  • Contributions: The review organizes methods around deep feature extraction, neural-network architecture, personal calibration, and devices or platforms.Coverage includes eye, face, and video inputs; supervised and unsupervised strategies; calibration; and RGB, infrared, and depth cameras across platforms.
  • Contributions: The paper summarizes preprocessing and postprocessing practices, converts gaze representations for fair comparison, and benchmarks state-of-the-art methods across datasets.It also collects typical algorithm implementations and public-dataset information.

2 GAZE ESTIMATION BACKGROUND

Gaze estimation has progressed from sensor-based and geometric methods toward appearance-based approaches using camera images. The paper places deep learning within this development and highlights the practical and modeling challenges that motivate it.

  • Appearance-based methods: Appearance-based methods directly map images to gaze using image pixels or learned features, while avoiding dedicated devices for geometric-feature detection.They remain challenging because facial appearance is complex.
  • Conventional methods: Conventional appearance-based methods use regression functions and subject-specific samples, creating time-consuming data-collection requirements.Reported approaches include linear interpolation, adaptive regression, and semi-supervised Gaussian-process regression.
  • Deep learning: Deep learning emerged to address head motion and subject differences, with CNNs, face images, videos, and methods for unconstrained conditions.Examples include asymmetric regression for extreme head pose and illumination.

3 DEEP GAZE ESTIMATION FROM APPEARANCE

The paper surveys deep learning-based gaze estimation through four perspectives spanning representation, model design, calibration, and deployment.

  • Survey scope: The survey covers deep feature extraction, neural-network architecture design, personal calibration, and devices and platforms.These perspectives organize the paper’s review of appearance-based gaze estimation methods.

3.1 Deep Feature from Appearance

Deep gaze estimation extracts representations from eyes, faces, or videos to handle identity, illumination, head pose, and temporal variation. Architectures combine CNNs with multiple inputs, attention, geometric constraints, recurrent models, and other learning strategies.

  • Overview: Feature extraction is difficult because identity and illumination complicate eye appearance, and feature quality affects gaze-estimation accuracy.The review groups mechanisms by eye images, face images, and videos.
  • Feature from Eye Images: Eye-image methods use CNNs to learn gaze features, sometimes combining both eyes or adding head-pose information.Multi-stream and asymmetric designs extract individual and joint representations from the two eyes.
  • Feature from Eye Images: Subject-invariant eye representations use unified gaze images, adversarial learning, autoencoders, geometric constraints, and GAN-based preprocessing.These approaches target variation from subjects, head motion, and environmental factors.
  • Feature from Face Images: Face-image methods exploit head-pose information but may suppress redundant regions through spatial weighting or combine face and eye streams.Some models separately estimate head and eyeball rotations before aggregating them into a gaze vector.
  • Feature from Videos: Video methods extract static CNN features per frame and feed them to an RNN, which captures temporal relations for gaze estimation.Other temporal cues include optical flow and eye-movement dynamics such as fixation, saccade, and smooth pursuit.

3.2 CNN Models

CNN-based gaze estimation methods are organized by learning strategy and architecture, spanning supervised, semi-/self-/unsupervised, multitask, and recurrent designs.

  • Learning strategies: Supervised CNNs learn a mapping from raw images to gaze directions using labeled training samples.Deeper CNN architectures generally achieve better performance, but training requires large-scale labeled datasets.
  • Learning strategies: Semi-supervised CNNs combine labeled and unlabeled images, using adversarial modules to improve performance on target subjects or datasets.The additional module distinguishes feature sources while the gaze network is supervised by labeled data.
  • Learning strategies: Self-supervised CNNs formulate auxiliary tasks, such as evaluating the reliability of gaze estimates from the two eyes, to guide training.A regression network supplies supervisory signals to an evaluation network, whose accuracy affects regression learning.
  • Learning strategies: Unsupervised CNNs train only with unlabeled data, using tasks such as gaze representation learning and gaze redirection despite the lack of ground truth.One approach feeds representation differences and an eye image into a pretrained gaze redirection network.
  • Network architectures: Multitask CNNs estimate related gaze features or auxiliary visual tasks simultaneously to provide inductive bias and improve generalization.Examples include landmark-gaze coefficients, eye segmentation, glint detection, gaze location, and visual-attention prediction.
  • Network architectures: Recurrent CNNs process frame sequences with recurrent networks to estimate gaze while incorporating temporal information.A unified CNN extracts features from each frame before a recurrent network outputs the gaze vector.

3.3 Calibration

Personal calibration adapts gaze estimation models to person- or domain-specific conditions, using supervised fine-tuning, unsupervised alignment, meta-learning, or user-unaware data.

  • Calibration via Domain Adaptation: Personal calibration treats the training set as a source domain and the test set as a target domain with unseen conditions.This framing connects person-specific calibration with domain adaptation.
  • Calibration via Domain Adaptation: Fine-tuning target-domain model components is a simple but effective calibration strategy.Reported variants fine-tune an SVM layer or encoder and decoder, and may learn a polynomial mapping function.
  • Calibration via Domain Adaptation: Unsupervised calibration uses unlabeled target samples to align extracted features or represent target domains locally.Adversarial calibration seeks domain-invariant features by requiring the extractor to confuse a source discriminator.
  • Calibration via Domain Adaptation: Meta-learning and metric-learning approaches support personalized gaze estimation with few-shot annotated samples for calibration.Meta-learning trains an adaptable network that becomes person-specific after target-person training.
  • Calibration via User-unaware Data Collection: User-unaware calibration collects samples during ordinary computer use to avoid requiring dedicated calibration data.Examples use mouse-click locations, saliency-derived losses, or predicted-gaze distributions for online adaptation.

3.4 Devices and Platforms

Appearance-based gaze estimation spans single and specialized camera configurations across computers, mobile devices, and head-mounted devices, with rectification reducing image variation.

  • Devices: Most gaze estimation systems use a single RGB camera, while others use multicamera, infrared, or RGBD configurations.These alternatives provide multi-view images, low-illumination capability, or depth information.
  • Devices: Multicamera systems concatenate features from different views to estimate gaze, including 2D gaze position on screens.Cameras may be mounted in glasses frames or below a display.
  • Platforms: Computers are the most typical platform, with cameras generally positioned above or below the screen.Research uses deeper networks, extra modules, and custom multicamera or RGBD devices.
  • Platforms: Mobile methods commonly estimate point of gaze because front cameras and limited computational resources make geometric calibration difficult.iTracker combines facial images, eye images, and a face grid encoding face position.
  • Data rectification: Data rectification rotates a virtual camera toward a reference point and transforms images to center, straighten, and scale the face.The transformation is represented as M = SR, combining rotation and scaling components.
  • Platforms: Head-mounted devices use near-eye cameras, including embedded RGB cameras and infrared cameras, to capture eye images.Multi-camera designs compensate for low-resolution near-eye imagery, while IR cameras support datasets such as MagicEyes.

3.5 Summary

The review identifies supervised and semi-/self-/unsupervised CNNs, domain-adaptation and user-unaware calibration, and the foundational role of MPIIGaze and GazeCapture.

  • 3.5 Summary: CNN-based gaze estimation research increasingly explores supervised and semi-/self-/unsupervised structures.The review also reports growing interest in calibration through domain adaptation and user-unaware data collection.
  • 3.5 Summary: MPIIGaze and GazeCapture are large-scale datasets introduced with the first CNN-based gaze-direction and point-of-gaze methods, respectively.Both datasets became widely used for evaluating later gaze estimation algorithms.

4 DATASETS AND BENCHMARKS

This section describes preprocessing, geometric gaze conversion, gaze-origin conversion, dataset summaries, and benchmark resources for appearance-based gaze estimation.

  • 4.1 Data Pre-processing: Face and eye detection crops aligned raw images to remove unnecessary background information before gaze regression.Facial landmarks support face and eye cropping, but the paper notes that no cropping protocol is standardized.
  • 4.1 Data Pre-processing: Data rectification uses a virtual-camera transformation to reduce variation from head pose, illumination, camera intrinsics, and head-roll freedom.The transformation is defined as M = SR, with perspective warping based on the original and new camera intrinsic matrices.
  • 4.4 Public Datasets and Benchmarks: The paper summarizes public datasets and benchmarks within-dataset performance using provided or reimplemented source code.Its dataset resources include a symbol table, dataset summary, and benchmark tables, while Fig. 15 shows gaze and head-pose distributions.
  • 4.2 Data Post-processing: 2D gaze targets are converted to 3D gaze directions by mapping screen coordinates into camera coordinates and normalizing the vector from gaze origin o to target t.The screen pose {Rs, Ts} computes t = Rs[u, v, 0]T + Ts; the origin is typically the face or eye center.
  • 4.2 Data Post-processing: 3D gaze directions are converted to screen points by intersecting the line of sight with the screen plane, then transforming the intersection into screen coordinates.The screen plane is defined from its normal vector and translation, and the intersection t is converted using {Rs, Ts}.
  • 4.2 Data Post-processing: Gaze-origin conversion bridges eye-centered and face-centered gaze estimates by finding the target point and recomputing the direction from a new origin.The new origin is obtained through 3D landmark detection.

4.3 Evaluation Metrics

Evaluation uses angular error for 3D gaze estimation and Euclidean distance for 2D gaze estimation, with both within-dataset and cross-dataset protocols.

  • 4.3 Evaluation Metrics: Angular error measures the accuracy of 3D gaze direction estimates.It compares actual gaze direction g with estimated gaze direction ĝ.
  • 4.3 Evaluation Metrics: Euclidean distance measures the accuracy of 2D gaze position estimates.It compares actual gaze position p with estimated gaze position.
  • 4.3 Evaluation Metrics: Within-dataset evaluation tests unseen subjects from the same dataset, while cross-dataset evaluation assesses performance across datasets.Within-dataset splits are subject-disjoint between training and test sets.

4.4 Public Datasets

The paper characterizes public gaze datasets spanning daily-life, laboratory, crowdsourced, indoor, and outdoor settings, alongside benchmark tables for cross-domain and 2D estimation.

  • 4.4 Public Datasets: The paper summarizes public gaze-estimation datasets and visualizes their gaze and head-pose distributions.Gaze360 does not provide head information.
  • 4.4 Public Datasets: MPIIGaze contains 213,659 images from 15 subjects collected during daily life without head-pose constraints, with both 2D and 3D gaze annotations.It also provides a standard evaluation set with 3,000 images per subject.
  • 4.4 Public Datasets: EyeDiap contains 94 laboratory video clips from 16 participants covering moving targets and static or freely moving head poses.Its stated disadvantage is a lack of illumination variation.
  • 4.4 Public Datasets: GazeCapture contains 2,445,504 crowdsourced mobile-device images from 1,474 participants under varied lighting and head motion.It lacks 3D target coordinates and is generally used for unconstrained 2D gaze-point estimation.
  • 4.4 Public Datasets: The benchmark resources include cross-domain and 2D gaze-estimation evaluations with dataset-specific protocol and adaptation information.The cross-domain table records source-image availability, target-image counts, and whether adaptation uses annotated or unannotated images.
  • 4.4 Public Datasets: Recent datasets extend coverage to accurate 3D gaze, wide head poses and distances, and indoor and outdoor environments.RT-Gene uses a dedicated eye-tracking device, while Gaze360 includes 238 subjects.

4.5 Benchmarks

The benchmark standardizes comparisons across 2D point-of-gaze and 3D gaze estimation by converting representations and gaze origins, then evaluates within-dataset, cross-dataset, and device-specific performance.

  • Benchmark setup: The benchmark converts between 2D point-of-gaze and 3D gaze because these methods estimate different gaze representations.It also converts gaze origins defined at eye centers or face centers before comparison.
  • Within-dataset evaluation: Face-image methods outperform eye-image methods despite higher computational costs, while typically maintaining inference speeds above 20 frames per second.This comparison concerns within-dataset evaluation.
  • Within-dataset evaluation: GazeTR-Hybird, CA-Net, and Gaze360 achieve better performance among face-image methods, with pre-training significantly improving GazeTR-Hybird and Gaze360.CA-Net combines facial and eye-image features without pre-training but uses a complex network.
  • Within-dataset evaluation: Large user-camera distances produce significant errors in Gaze360 and RT-Gene, whereas other environments usually yield approximately 5° error.The paper attributes these differences to low-resolution images in the large-distance datasets.
  • Cross-dataset evaluation: Cross-dataset evaluation compares DE → DM, DE → DD, DG → DM, and DG → DD using ETH-XGaze and Gaze360 as training datasets.The target datasets are MPIIFaceGaze and EyeDiap.
  • Cross-dataset evaluation: Domain-adaptation methods generally outperform source-only methods within target domains by using target images, while source-only methods can generalize directly across multiple domains.CSA is source-free, and PureGaze-FT requires five images per person with annotations for fine-tuning.
  • 2D PoG estimation: AFF-Net and EFE show the best 2D point-of-gaze performance among compared methods, while converted gaze-direction results are accurate on EyeDiap.AFF-Net also performs best on MPIIGaze in the converted comparison.

5 CONCLUSIONS AND FUTURE DIRECTIONS

The survey consolidates deep learning-based appearance gaze estimation methods, datasets, and benchmarks, and identifies robustness, calibration efficiency, and interpretability as future directions.

  • Conclusions: The survey reviews deep learning-based appearance gaze estimation across feature extraction, network architecture, personal calibration, and device or platform.It also summarizes public datasets and provides benchmarks for comparing state-of-the-art algorithms.
  • Future directions: The authors propose extracting environment-invariant gaze features that remain accurate across subjects, head poses, and environments.This is presented as a direction for improving robustness.
  • Future directions: Fast, simple calibration remains an open direction because longer calibration improves accuracy but increases calibration time.The paper frames this as a tradeoff between system performance and calibration time.
  • Future directions: Interpreting learned features is another future direction because deep learning approaches often function as black-box gaze estimators.The authors expect interpretation to provide insight into learned gaze-estimation features.
Loading 2104.12668v2…