PaperlayerView exact revision on arXiv

Source-linked AI summary

MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation

Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

arXiv:1711.09017v1cs.CV

TL;DR

Unconstrained gaze estimation lacked realistic datasets and cross-dataset evaluation, making generalisation beyond laboratory conditions unclear. The paper introduces MPIIGaze, evaluates methods across three datasets, and proposes GazeNet, which improves the state of the art in the most challenging cross-dataset setting.

  • Problem

    Existing gaze datasets were collected under laboratory conditions, and gaze-estimation methods had not been properly evaluated across different datasets.

  • Method

    The paper constructs MPIIGaze from long-term everyday laptop use, adds facial and pupil annotations, evaluates methods across three datasets, and develops a 16-layer VGG-based GazeNet.

  • Results

    GazeNet improves the state of the art by 22% on MPIIGaze and 8% on EYEDIAP in the most difficult cross-dataset evaluation.

  • Takeaways & Limitations

    Cross-dataset evaluation reveals that unconstrained gaze estimation is substantially harder than within-dataset evaluation and that realistic training data is important.

  • Takeaways & Limitations

    The usefulness of eye-shape and pupil-centre features may be limited for unconstrained gaze estimation on real-world images.

Abstract

from arXiv · show

Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze that contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves the state of the art by 22% percent (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.

1 INTRODUCTION

The paper addresses limited cross-dataset evidence and laboratory bias in gaze estimation by introducing MPIIGaze, evaluating existing methods, and proposing GazeNet.

  • Motivation: Current gaze datasets were collected under controlled laboratory conditions, limiting appearance and illumination variability and leaving cross-dataset generalisation insufficiently evaluated.These limitations create a risk of dataset bias and leave performance in real-world settings unclear.
  • MPIIGaze Dataset: 213,659 images from 15 laptop users were collected over several months during daily life using experience sampling to capture realistic gaze, appearance, and illumination variation.The dataset also includes 37,667 images with annotations for facial landmarks and pupil centres.
  • Evaluation: The authors evaluate state-of-the-art methods across MPIIGaze, EYEDIAP, and UT Multiview to study gaze range, illumination, and personal differences.The analysis examines the performance gap between within-dataset and cross-dataset evaluation conditions.
  • MPIIGaze Dataset: MPIIGaze samples show substantial variation in recording place and time, eye appearance, and illumination, including directional light and shadows.The figure contrasts these samples with images from UT Multiview, EYEDIAP, and Columbia.
  • GazeNet: GazeNet is a deep appearance-based method using a 16-layer VGG convolutional neural network that improves cross-dataset performance by 22% on MPIIGaze and 8% on EYEDIAP.The reported improvements concern the most difficult cross-dataset evaluation.

2 RELATED WORK

Related work spans model-based and appearance-based gaze estimation, person-independent evaluation, and datasets with varying coverage of participants, poses, illumination, annotations, and recording duration.

  • Gaze Estimation Methods: Model-based methods infer gaze from geometric eye features, including corneal reflections or eye shape such as pupil centres and iris edges.Their accuracy depends on accurate feature detection, high-resolution images, and relatively homogeneous illumination.
  • Gaze Estimation Methods: Appearance-based methods directly regress from eye images to 3D gaze directions without explicit feature-point detection.They can handle low-resolution images and long distances but require more person-specific training data to cover appearance variation.
  • Person-Independent Evaluation: Earlier person-independent studies evaluated models across participant groups, but training and test data still came from the same dataset.The paper identifies cross-dataset evaluation as a more practical and challenging generalisation setting.
  • Datasets: Existing gaze datasets differ in participants, head poses, gaze targets, illumination, annotations, dataset size, collection duration, and 3D ground truth.These dimensions determine whether datasets support meaningful cross-dataset evaluation.

3 THE MPIIGAZE DATASET

MPIIGaze is a large, long-term dataset collected during everyday laptop use, designed to capture realistic variation in gaze, head pose, illumination, and facial appearance. It also provides 3D annotations and manually verified facial landmarks to support cross-dataset evaluation and related face-analysis tasks.

  • Dataset Collection: MPIIGaze was recorded during participants’ daily laptop use over several months to cover varied locations, times, illuminations, and eye appearances.Participants were not otherwise constrained in how or where they used their laptops.
  • Facial Landmark Annotation: 37,667 images received manual facial-landmark annotations, including eye corners, mouth corners, and pupil centres, enabling evaluation of alignment errors and other face-related tasks.The landmark process combined automatic detection with manual correction, while pupil centres were annotated manually and were not used as method input.
  • Dataset Collection: Every 10 minutes, laptop software prompted participants to fixate on a random sequence of 20 on-screen positions, providing ground-truth gaze targets.On-screen positions were converted to 3D camera-coordinate positions using camera calibration because laptops differed in screen size and resolution.
  • Dataset Collection: 213,659 images from 15 participants were collected over periods ranging from 9 days to 3 months, with substantial variation in recording times and illumination.Participants included five people wearing glasses and had varied iris colours; image counts ranged from 1,498 to 34,745 per participant.
  • Dataset Characteristics: MPIIGaze covers gaze and head-angle distributions for comparison with UT Multiview and EYEDIAP, while UT Multiview offers broad pose-space coverage under controlled lighting.The distributions are shown in degrees in the normalised head-and-gaze space.
  • Dataset Characteristics: MPIIGaze exhibits larger eye-region appearance variation than UT Multiview and EYEDIAP, particularly for participants wearing glasses.The comparison uses samples from one person at roughly matching gaze directions across datasets.
  • Facial Landmark Annotation: 85% of images had no detected facial-landmark error, whereas pupil detection was evaluated using RMS error normalised by inter-eye distance.A normalised RMS error of 0.01 roughly corresponds to pupil size.

4 METHOD

GazeNet forms a practical, multimodal gaze-estimation pipeline by detecting facial structure, estimating head pose, normalising eye images, and mapping eye appearance plus head angle to gaze angles.

  • Face Alignment and 3D Head Pose Estimation: GazeNet uses facial landmarks from a monocular RGB image to estimate 3D head pose with a generic mean facial shape model.The model comprises six 3D eye and mouth-corner landmarks and avoids assuming accurate head poses.
  • Eye Image Normalisation: The normalisation conversion is M = SR, where S scales depth so the eye lies at distance d_n and R rotates the camera toward the eye.The same process is applied to the right and left eyes using their corresponding eye positions.
  • Eye Image Normalisation: Eye-image normalisation compensates for camera pose and scale by warping each eye into a normalised coordinate system with its centre at a fixed position.The conversion uses a rotation and scaling matrix, followed by perspective image warping.
  • GazeNet Architecture: The CNN maps a 2D head angle h and eye image e to 2D gaze angles g in the normalised space.This formulation avoids requiring training data to model the angle of convergence between the eyes.
  • GazeNet Architecture: GazeNet extends a 16-layer VGGNet with 13 convolutional layers, two fully connected layers, five max-pooling layers, and head-pose input at fc6.It uses 60 × 36 grayscale eye images and sums individual L2 losses for predicted and true gaze-angle vectors.

5 EXPERIMENTS

The experiments evaluate GazeNet across datasets and persons while probing gaze range, illumination, personal appearance, image resolution, eye usage, head pose, and pupil-centre information.

  • Experimental Scope: The evaluation covers cross-dataset and cross-person gaze estimation together with several factors affecting unconstrained performance.The study explicitly examines gaze ranges, illumination conditions, and personal appearance.
  • Experimental Scope: The experiments also test how image resolution, using both eyes, head pose, and pupil-centre information influence gaze-estimation performance.

Baseline Methods

The study compares GazeNet with neural, tree-based, instance-based, linear, support-vector, and shape-based gaze-estimation baselines using matched dataset subsets and cross-dataset tests.

  • Compared Methods: The baselines include MnistNet, Random Forests, kNN, Adaptive Linear Regression, linear SVR, and the shape-based EyeTab method.EyeTab estimates gaze by fitting a fixed-diameter limbus model to detected iris edges.
  • Datasets: Experiments use 1,500 left-eye and 1,500 right-eye images per participant, with oversampling for one participant having fewer face images.The manually annotated version of this subset is called MPIIGaze+, while the corresponding unannotated subset is MPIIGaze.
  • Datasets: Cross-dataset testing uses MPIIGaze, MPIIGaze+, or EYEDIAP after training on UT Multiview or SynthesEyes.

Evaluation Procedure

Cross-dataset evaluation trains methods on UT Multiview or SynthesEyes and tests them on MPIIGaze, MPIIGaze+, or EYEDIAP, while cross-person evaluation uses leave-one-person-out validation.

  • Cross-Dataset Evaluation: Each method is trained on UT Multiview or SynthesEyes and tested on MPIIGaze, MPIIGaze+, or EYEDIAP for cross-dataset evaluation.UT Multiview is selected because it covers the largest head- and gaze-angle area among the compared real datasets; SynthesEyes matches its angle ranges.
  • Cross-Person Evaluation: Cross-person evaluation uses leave-one-person-out cross-validation.

5.1 Performance Evaluation

Cross-dataset evaluation shows substantial difficulty on real-world MPIIGaze, while GazeNet achieves the lowest errors and remains ahead of prior baselines. Cross-person evaluation improves all methods, but GazeNet still outperforms the second-best method.

  • Cross-Dataset Evaluation: GazeNet achieved the lowest cross-dataset error on MPIIGaze and EYEDIAP, at 10.8 and 9.6 degrees, respectively.
  • Cross-Dataset Evaluation: 22% (3.1 degrees) improvement on MPIIGaze and 8% (0.9 degrees) on EYEDIAP exceeded the state-of-the-art method.Both gains were statistically significant with p < 0.01 using paired Wilcoxon signed rank tests.
  • Cross-Dataset Evaluation: Reducing GazeNet input images to 15 × 9 pixels produced 11.4 degrees error on MPIIGaze, still outperforming other baseline methods.
  • Cross-Dataset Evaluation: Manual facial landmarks reduced GazeNet error from 10.8 to 9.8 degrees, identifying face detection and landmark alignment as a dominant practical error factor.
  • Cross-Person Evaluation: All methods performed better in cross-person than cross-dataset evaluation, and GazeNet outperformed MnistNet by 13% with 5.5 degrees mean error.The cross-person evaluation used leave-one-person-out validation.
  • Cross-Person Evaluation: Domain-specific training improved appearance-based gaze estimation, while noisy landmark detections produced no noticeable improvement on MPIIGaze+ in cross-person testing.

5.2 Key Challenges

The analyses identify gaze-range mismatch, illumination variation, and personal appearance as major sources of unconstrained gaze-estimation difficulty. Matching target gaze ranges and adding varied illumination improve performance, but real-world person-independent estimation remains challenging.

  • Differences in Gaze Ranges: 18% improvement, from 9.8 to 8.0 degrees, resulted from restricting training data to the target domain and gaze range.
  • Differences in Gaze Ranges: Training on the whole UT Multiview dataset was more robust across gaze directions than training on a subset, despite differing gaze-direction distributions.
  • Differences in Illumination Conditions: Synthetic training improved error from 8.0 to 7.3 degrees under matched gaze-range conditions, showing the relevance of illumination variation in training data.
  • Differences in Illumination Conditions: Lighting-direction analysis compared errors across left-right face intensity differences for models trained on UT subset, synthetic subset, and MPIIGaze+.
  • Personal Appearance: Person-specific evaluation still showed that in-the-wild settings are challenging, even when training and testing data came from the same person.
  • Personal Appearance: Participants wearing glasses had the worst cross-dataset performance in the participant-level analysis.The identified participants were P5, P8, and P10.

5.3 Further Analyses

Further analyses show that resolution, bilateral eye information, head pose, and pupil centres affect performance unevenly. Late fusion of both eyes helps, whereas higher resolution, head-pose features, and pupil-centre inputs provide limited or non-monotonic benefits in the tested settings.

  • Image Resolution: Higher resolution did not always improve performance; the 30 × 18 model outperformed the highest-resolution model in the cross-dataset analysis.
  • Use of Both Eyes: Averaging predictions from both eyes achieved 9.8 degrees error, while geometrically merging their 3D gaze vectors reduced error to 7.2 degrees.The vector-merging result had a 1.4-degree standard deviation.
  • Use of Both Eyes: The 7.2-degree late-fusion result indicates potential for more sophisticated methods that combine information from both eyes.
  • Head Pose and Pupil Centres: Adding head-pose features had only a marginal effect in cross-dataset evaluation, likely because MPIIGaze head-pose variation was limited to near-frontal cases.
  • Head Pose and Pupil Centres: Pupil-centre information was not used in GazeNet because shape-based methods performed relatively poorly in cross-dataset and cross-person evaluations.

6 DISCUSSION

The paper frames unconstrained gaze estimation as practically relevant but substantially harder than domain-specific evaluation. Its analyses emphasize data coverage and correct 3D geometry, while indicating limited usefulness of shape-based cues in real-world cross-dataset settings.

  • Discussion: Cross-dataset evaluation reduced performance by up to 69%, from 5.4 to 9.1 degrees, compared with person-independent domain-specific evaluation.
  • Discussion: MPIIGaze exposes challenges that prior work often avoided by restricting gaze or head-pose ranges, illumination, recording duration, or 3D gaze evaluation.
  • Discussion: Unconstrained gaze estimation requires a 3D formulation and proper handling of 3D geometry, including head pose.
  • Discussion: The shape-based method achieved 47.1 degrees mean error in cross-dataset evaluation, while detected pupil-centre inputs yielded only a small CNN improvement.

7 CONCLUSION

The paper addresses simplifications in unconstrained gaze estimation with an everyday-life dataset, extensive cross-dataset evaluation, and a deep appearance-based method. Its proposed method improves performance in the most challenging cross-dataset evaluation on MPIIGaze.

  • MPIIGaze captures substantial variation in eye appearance and illumination through collection during everyday laptop use over several months.The dataset also includes manually annotated facial landmarks for a large image subset, supporting cross-dataset evaluations.
  • Extensive evaluation of appearance- and model-based methods demonstrates the need for new methods for unconstrained gaze estimation.The evaluations cover three current datasets and expose challenges in generalising beyond prior simplified conditions.
  • 22% improvement is achieved by the proposed deep convolutional neural-network appearance-based method in the most challenging cross-dataset evaluation on MPIIGaze.
  • The evaluations provide a detailed account of the state of the art and guide future research on appearance-based gaze estimation.
Loading 1711.09017v1…