PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Adaptive Gaze Estimation

Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, Jan Kautz

arXiv:1905.01941v2cs.CV

TL;DR

Person-independent gaze estimation remains too inaccurate for some high-accuracy applications, while personalized CNNs typically require many examples and overfit with few samples. FAZE combines a disentangling rotation-aware encoder-decoder representation with a meta-learned adaptable estimator, achieving 3.18° on GazeCapture with 3 calibration samples and state-of-the-art personalized errors on two datasets.

  • Problem

    Inter-person anatomical differences limit person-independent gaze accuracy, while person-specific CNN training needs thousands of images and overfits with few calibration samples.

  • Method

    FAZE learns a rotation-aware latent representation disentangling gaze, head pose, and appearance, then trains an adaptable gaze estimator using meta-learning.

  • Results

    19% improvement, from 3.91° to 3.18°, is reported on GazeCapture over prior work using 3 calibration samples, with state-of-the-art personalized errors on GazeCapture and MPIIGaze.

  • Takeaways & Limitations

    FAZE enables practical high-accuracy personalized gaze estimation from as few as 3 calibration points.

  • Takeaways & Limitations

    Embedding consistency is applied only to intra-subject image pairs, rather than enforcing identical gaze features across people.

Abstract

from arXiv · show

Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks. Yet there is a need to lower gaze errors further to enable applications requiring higher quality. Further gains can be achieved by personalizing gaze networks, ideally with few calibration samples. However, over-parameterized neural networks are not amenable to learning from few examples as they can quickly over-fit. We embrace these challenges and propose a novel framework for Few-shot Adaptive GaZE Estimation (FAZE) for learning person-specific gaze networks with very few (less than or equal to 9) calibration samples. FAZE learns a rotation-aware latent representation of gaze via a disentangling encoder-decoder architecture along with a highly adaptable gaze estimator trained using meta-learning. It is capable of adapting to any new person to yield significant performance gains with as few as 3 samples, yielding state-of-the-art performance of 3.18 degrees on GazeCapture, a 19% improvement over prior art. We open-source our code at https://github.com/NVlabs/few_shot_gaze

1. Introduction

FAZE addresses the accuracy limits and few-shot overfitting challenges of personalized gaze estimation by combining a disentangled rotation-aware representation with meta-learned adaptation. It achieves substantial gains with as few as 3 calibration samples.

  • Motivation: 4.3° is the lowest reported person-independent gaze error, yet it still limits high-accuracy applications.At 60 cm, this corresponds to 4.7 cm of positional error.
  • Approach: FAZE trains AdaGEN with meta-learning so it can adapt into a robust person-specific gaze estimator using very little calibration data.The framework combines learned latent features with an adaptable gaze estimator trained for few-shot personalization.
  • Motivation: Few-shot personalization is difficult because CNNs require thousands of subject-specific images and overfit when trained on very few samples.Classical model-based methods can personalize with 9 or fewer samples but are not robust to uncontrolled image variation.
  • Approach: FAZE learns a rotation-aware latent gaze representation with a disentangling encoder-decoder that separates gaze, head pose, and appearance.The framework enforces equivariance by decoding rotated latent codes into images of the same person with different gaze directions.
  • Results: 19% improvement, from 3.91° to 3.18°, is achieved on GazeCapture over prior work using 3 calibration samples.On MPIIGaze, the reported improvement is 13%, from 3.94° to 3.42°, with 3 calibration samples.
  • Results: 3.14° with k = 9 is the reported state-of-the-art performance on MPIIGaze, with improvements over existing methods for 1 ≤ k ≤ 256.The contribution summary reports consistent gains across the evaluated calibration sizes.

2. Related Work

Prior work improved person-independent gaze estimation through stronger architectures, richer inputs, normalization, and intermediate representations, but high errors and few-shot overfitting remain challenges. FAZE builds on learned equivariance and meta-learning to address personalized gaze estimation from few examples.

  • Gaze Estimation: Person-independent gaze error has fallen from 6.3° to 4.3° on competitive within-MPIIGaze evaluations.These gains came from advances including complex CNNs, face and multimodal inputs, eye-specific handling, pose robustness, normalization, and ensembles.
  • Gaze Estimation: Person-independent errors remain insufficient for many applications, while person-specific CNN models require thousands of training images per subject.Existing approaches adapt CNN features or points-of-regard to person-specific ones when few samples are available.
  • Learned Equivariance: Transforming encoder-decoder architectures improve learned equivariant mappings between inputs, latent features, and labels in several vision tasks.FAZE applies this idea to noisy real-world gaze and head-orientation phenomena.
  • Few-shot Learning: Few-shot learning targets new tasks from very few examples, but highly over-parameterized deep networks can overfit in this setting.Meta-learning methods address this by learning across unique but related tasks, motivating their use for person-specific gaze networks.

3. Method

FAZE learns a generalizable, rotation-aware gaze representation by disentangling appearance, gaze direction, and head pose, then uses meta-learning to adapt gaze estimators to new people with few calibration samples.

  • Framework overview: FAZE adapts person-specific gaze estimators to new people using very few calibration samples, with k ≤9 as the target setting.The framework is organized around learning features, meta-learning an adaptable estimator, and adapting it to each new person.
  • Framework overview: The first stage learns a generalizable latent embedding that captures gaze-direction information while retaining person-specific aspects and robustness to appearance variation.The embedding is intended to leverage eye-region observations across people, head poses, and gaze configurations.
  • DT-ED representation: DT-ED trains on paired images of the same person by rotating latent codes and decoding the transformed representation to reconstruct the paired image.The architecture uses known gaze and head rotations, with reconstruction enforcing consistency between transformed codes and the target image.
  • DT-ED representation: DT-ED disentangles appearance, gaze direction, and head pose into separate latent codes and explicitly rotates the gaze and head-pose codes.The gaze and head-pose codes are dimensioned to support explicit 3D rotation-matrix operations.
  • DT-ED representation: The representation is demonstrated by frontalizing samples and reconstructing images across 15 predefined gaze and head-pose orientations.The reported reconstructions show disentangled gaze and head-pose factors and support eye-region frontalization and redirection from monocular RGB input.
  • Training objectives: Training combines pixel-wise L1 reconstruction, gaze regression, and an embedding-consistency objective applied to frontalized intra-person gaze features.The consistency term uses only intra-subject pairs to preserve inter-subject anatomical differences, and its weight is increased gradually during training.
  • Training objectives: A gaze estimator maps the learned gaze code to predicted gaze direction through an MLP trained with a gaze-direction loss.This estimator is the prediction component built on top of the learned latent representation.

4. Implementation Details

The implementation uses normalized eye images, DenseNet-based transforming encoder-decoder and MLP components, and specified optimization settings across GazeCapture and MPIIGaze evaluations.

  • Preprocessing: The preprocessing normalizes inputs to a common virtual camera with the head upright and a shared reference point.The procedure includes virtual-camera rotation, tilt, and forward translation.
  • Network configurations: The transforming encoder-decoder uses DenseNet, with latent dimensions 64 for appearance, 3 × 2 for gaze, and 3 × 16 for head pose.The encoder and decoder can use any CNN architecture, but DenseNet is selected here.
  • Network configurations: The gaze estimator is a 64-neuron SELU MLP that outputs 3-dimensional unit gaze direction vectors.Meta-learning uses SGD and Adam settings specified for the inner and outer updates, respectively.
  • Optimization: Final fine-tuning runs for 1000 steps for every calibration size and person because extra ground truth is unavailable during standard calibration.The transforming encoder-decoder is trained for 50 epochs with a base learning rate of 5 × 10^-5 and l2 regularization of 10^-4.
  • Datasets: GazeCapture training uses 993 people and approximately 1.7M samples, while evaluation uses the last 500 entries from 109 subjects.Subjects with at least 400 samples support training diversity; evaluated subjects have at least 1000 samples.
  • Datasets: MPIIGaze evaluation uses 15 subjects with approximately 2500 samples each, reserving the last 500 images for final evaluation.Calibration samples are randomly selected from the remaining images.

5. Results

Experiments examine adaptation, representation, loss design, and comparisons against CNN-plus-MAML and person-specific baselines. FAZE improves errors across calibration settings and datasets, with gains supported by ablations and statistically significant comparisons.

  • MAML vs. Finetuning: MAML adaptation lowers mean error from 7.17° without adaptation to 6.61° with one sample and 5.38° with 32 samples.Naive AE fine-tuning overfits severely and exceeds the person-independent baseline for k ≤3.
  • Feature representation: 4.87° versus 5.62° at k = 9 shows lower error for DT-ED (MAML) than AE (MAML).The gain remains consistent across calibration sizes; DT-ED differs by rotating latent codes before decoding.
  • Embedding consistency: Person-independent embedding consistency increases mean errors and performs worse than using only reconstruction loss.Triplet loss, which also maximizes inter-person differences, significantly increases error.
  • CNN + Meta-Learning: FAZE outperforms competitive DenseNet and VGG-16 CNN-plus-MAML baselines with statistical significance on MPIIGaze.The comparison evaluates features learned directly from the cross-person gaze objective against DT-ED features.
  • State-of-the-art comparison: FAZE achieves statistically significantly better mean errors than existing state-of-the-art methods across 1 ≤ k ≤256, with narrower trial error bands.The narrower bands indicate more consistent performance across calibration-sample choices.
  • State-of-the-art comparison: 20.5% or 0.8° improvement over at k = 4 is reported on the GazeCapture test subset.On MPIIGaze, FAZE requires four calibration samples to compete with at k = 256.

6. Conclusion

The paper presents FAZE as a practical deep-learning approach for personalized gaze estimation from very few calibration samples. It reports significant gains over prior methods on GazeCapture and MPIIGaze.

  • Conclusion: FAZE combines a disentangling encoder-decoder with meta-learning to learn person-specific gaze networks from as few as k = 3 calibration points.The latent representation includes gaze, head pose, and appearance factors.
  • Conclusion: The authors report significant margins over state-of-the-art approaches and the lowest reported personalized gaze errors on both benchmark datasets.The conclusion states this result for GazeCapture and MPIIGaze.

A. Implementation Details

The supplementary material adds preprocessing and transforming encoder-decoder configuration details, along with additional evaluations and sensitivity analyses.

  • Supplementary implementation details: The supplementary material provides further dataset preprocessing and DT-ED configuration details, with reference implementations available as open-source software.It also includes additional GazeCapture ablations, within-MPIIGaze leave-one-person-out results, and configuration sensitivity.

A.1. Data Pre-processing

The preprocessing pipeline normalizes face and eye imagery using detected landmarks, a reference 3D face model, and PnP-based head-pose estimation before training.

  • Face detection and facial-landmark detection use state-of-the-art open-source implementations.
  • The Surrey Face Model and selected eye-corner and nose landmarks support PnP-based head-pose estimation.
  • Each input is a single image containing both eyes, normalized around the mean of the two inner eye corners.
  • Normalized camera parameters use a 1300mm focal length and 600mm face distance to produce 256 × 64 training patches.
  • The encoder-decoder uses DenseNet blocks with instance normalization and leaky ReLU activations, while its decoder replaces convolutions and pooling with deconvolutions.The DenseNet configuration uses growth-rate 32, four dense blocks, and four composite layers per block.

B. Additional Results

Additional experiments report ablation results on GazeCapture and evaluate FAZE within the MPIIGaze dataset.

  • The supplementary material provides additional results for the ablation study on the GazeCapture test partition.
  • The additional experiments also evaluate FAZE’s within-dataset performance on MPIIGaze.
  • These results extend the paper’s analysis beyond its main reported ablation setting.

B.1. Ablation Study on GazeCapture

On GazeCapture, the ablations reproduce the main trends: transforming encoder-decoder features and MAML improve few-shot personalization, while naive fine-tuning overfits at very low calibration counts.

  • B.1. Ablation Study on GazeCapture: The GazeCapture ablation observes the same trends as the MPIIGaze ablation study.
  • B.1. Ablation Study on GazeCapture: Transforming encoder-decoder features produce latent representations better suited for gaze estimation than naive encoder-decoder features.
  • B.1. Ablation Study on GazeCapture: Meta-learning an adaptable estimator yields significant few-shot accuracy gains over naively fine-tuning a person-independent estimator.
  • B.1. Ablation Study on GazeCapture: Naive fine-tuning leads to overfitting when very few calibration samples are available.
  • B.1. Ablation Study on GazeCapture: Embedding consistency and latent-representation gaze losses contribute to training the transforming encoder-decoder, while preserving inter-person differences improves accuracy.

B.2. Within-MPIIGaze Performance

Within the MPIIGaze leave-one-subject-out protocol, FAZE improves substantially over the prior method, while cross-dataset training benefits from greater subject diversity.

  • B.2. Within-MPIIGaze Performance: Figure 7 is an ablation study on GazeCapture test data covering MAML, transforming encoder-decoder features, training losses, and embedding-consistency variants.
  • B.2. Within-MPIIGaze Performance: 3.88° is the gaze error achieved by FAZE with 9 calibration samples under the within-MPIIGaze protocol.
  • B.2. Within-MPIIGaze Performance: 17% improvement over Liu et al.’s 4.67° result is obtained with 9 calibration samples.
  • B.2. Within-MPIIGaze Performance: Within-MPIIGaze training performs worse than training with GazeCapture.The text attributes this to GazeCapture’s larger training-subject diversity: 993 versus 14 in the leave-one-out MPIIGaze setting.

C. Sensitivity Analysis

FAZE’s sensitivity analysis examines latent gaze-code dimensionality and normalization choices, with experiments comparing alternative settings across the two datasets.

  • FAZE evaluates design parameters empirically to determine their optimal values.The analysis covers latent gaze-code dimensionality and normalization.
  • Dimension: Fg = 2 is empirically optimal for both datasets when the latent gaze code has dimensions 3 × Fg.The comparison evaluates Fg ∈ {16, 3, 2} while fixing appearance and head-pose code dimensions at 64 and 16.
  • Latent Gaze Code: Figure 8 compares gaze errors under within-MPII leave-one-person-out training and GazeCapture training followed by testing on one MPIIGaze subject.
  • Normalization: Normalizing along the Fg dimension produces lower gaze errors for GazeCapture and equivalent errors for MPIIGaze compared with normalizing along the 3 dimension.The alternatives use ℓ2 norms computed along either the Fg or 3 dimension.
  • Sensitivity Analysis: Figures 9 and 10 examine FAZE performance across latent gaze-code dimensions and normalization axes, respectively.
Loading 1905.01941v2…