PaperlayerView exact revision on arXiv

Source-linked AI summary

A Differential Approach for Gaze Estimation

Gang Liu, Yu Yu, Kenneth A. Funes Mora, Jean-Marc Odobez

arXiv:1904.09459v3cs.CV

TL;DR

Appearance-based gaze estimation is limited by subject-specific eye and axis variation, and calibration may be needed. The paper trains a differential network to predict gaze differences between same-subject eye images and uses calibration references to infer gaze. Across its reported experiments, the differential method is more robust and stable across sessions than linear adaptation, with some additional runtime cost relative to a baseline CNN.

  • Problem

    Direct gaze regression is limited by subject-dependent eye and visual-to-optical-axis variation, contributing to accuracy around 5 to 6 degrees and high inter-person variance.

    Source: §1 · p. 1

  • Method

    A differential neural network predicts gaze differences between two eye images, which are combined with calibration reference gazes to estimate a novel image’s gaze.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Experiments on three public datasets report that the differential method is more robust and stable across sessions than linear adaptation, at some additional runtime cost relative to a baseline CNN.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The approach supports subject-specific gaze inference using a few calibration samples with a unified, person-independent differential model.

    Source: §4.1 · p. 4

    The approach does not provide additional robustness to geometric domain shifts, and cross-dataset comparisons can be limited by coordinate-system and preprocessing differences.

    Source: §5.4 · p. 6

Abstract

from arXiv · show

Non-invasive gaze estimation methods usually regress gaze directions directly from a single face or eye image. However, due to important variabilities in eye shapes and inner eye structures amongst individuals, universal models obtain limited accuracies and their output usually exhibit high variance as well as biases which are subject dependent. Therefore, increasing accuracy is usually done through calibration, allowing gaze predictions for a subject to be mapped to his/her actual gaze. In this paper, we introduce a novel image differential method for gaze estimation. We propose to directly train a differential convolutional neural network to predict the gaze differences between two eye input images of the same subject. Then, given a set of subject specific calibration images, we can use the inferred differences to predict the gaze direction of a novel eye sample. The assumption is that by allowing the comparison between two eye images, annoyance factors (alignment, eyelid closing, illumination perturbations) which usually plague single image prediction methods can be much reduced, allowing better prediction altogether. Experiments on 3 public datasets validate our approach which constantly outperforms state-of-the-art methods even when using only one calibration sample or when the latter methods are followed by subject specific gaze adaptation.

1 INTRODUCTION

Appearance-based gaze models face subject-dependent variation and limited accuracy, while calibration can require substantial personal data or be sensitive to few references. The paper proposes differential gaze prediction and reports mid-level fusion as the best of the compared architectures.

1 INTRODUCTION

  • Even with DNN regressors, gaze accuracy remains limited to around 5 to 6 degrees, with high inter-person variance, due to head-pose dependencies, large eye-shape variability, and subtle appearance changes –.Source: §1 · p. 1
  • Person-specific models and fine-tuning can require substantial personal data, while few-reference linear regression or SVR can be sensitive to environment and reference count.Too few references can produce worse performance than no calibration.Source: §1 · pp. 1–2
  • A linear adaptation baseline learns a mapping from pretrained-network gaze predictions to ground-truth gaze and is reported to outperform the SVR method of.The paper also presents a unified, person-independent differential model for subject-specific inference from a few annotated references.Source: §1 · p. 2
  • A differential network predicts gaze differences between two eye images, then uses reference gazes to infer a novel sample’s gaze.The authors hypothesize that comparing images from the same person alleviates person-dependent errors such as shape and alignment differences.Source: §1 · p. 2
  • Mid-level fusion performs better than early or late fusion for differential gaze prediction.The paper compares architectures that combine the two images or their feature maps at different levels.Source: §1 · p. 2

2 RELATED WORKS

The related work covers appearance-based estimation, person-dependent calibration, and siamese networks. It positions differential gaze estimation as a distinct way to calibrate from reference images.

2 RELATED WORKS

  • The paper situates its work across appearance-based gaze estimation, person-dependent calibration, and siamese-network approaches.These are the three strands identified in the related-work overview.Source: §2 · p. 2

2.1 Appearance-based gaze estimation

  • Appearance-based methods use eye or face images directly; neural networks have shown good generalization and can handle varied imaging conditions.Prior examples include networks using eye images, combined eye and face inputs, full-face images, and simulated eye images.Source: §2.1 · p. 2

2.2 Person dependent calibration

  • Prior calibration approaches include few-sample linear regression, SVR, fine-tuning, and user-interaction data, with stated robustness or data-use constraints.The paper identifies these approaches as not addressing calibration through gaze differences between reference images.Source: §2.2 · p. 3

2.3 Siamese network

  • Siamese networks use parallel shared-weight branches on image pairs; prior head-pose work still predicts pose from a single image at test time.A closer few-shot relation-network approach addresses image classification and does not adapt using reference samples.Source: §2.3 · p. 3

3 BASELINE CNN APPROACH AND LINEAR ADAP-

The section establishes a person-independent CNN baseline and a linear calibration model for correcting person-specific gaze bias. It also describes the training setup and the observed relationship between network predictions and labeled gaze.

TATION

  • The section introduces a standard CNN, analyzes its bias for unknown individuals, and presents linear adaptation as a baseline correction.The adapted prediction is defined as gad(I) = A gp(I) + B.Source: TATION · p. 3, §3.2 · p. 3

3.1 Gaze estimation with CNN

  • A gaze-estimation network whitens 48 × 72 eye images (three channels), processes them through convolutional and fully connected layers, and regresses a two-dimensional gaze direction using an L1 loss.Smaller MPIIGaze images are bilinearly up-sampled; EYEDIAP images are randomly cropped around eyes to 48 × 72. Training uses Adam at an initial learning rate of 0.001, halved after each epoch, for 10 epochs with mini-batches of 128.Source: §3.1 · p. 3

3.2 Bias analysis and baseline linear adaptation method

  • Within a person, network gaze predictions usually show bias relative to labeled gaze, with a usually linear relationship that can be fitted using calibration samples.The described calibration set usually contains 9 to 25 points; the linear parameters are estimated by least mean square error optimization.Source: §3.2 · p. 3

4 PROPOSED DIFFERENTIAL APPROACH

The proposed method predicts gaze differences between two images of the same eye, then combines those differences with calibrated reference gazes to estimate a test gaze. Its network, training, optional adaptation, and reference weighting implement this approach.

4.1 Approach overview.

  • The differential network predicts gaze differences between two same-eye images, addressing subject-specific bias and within-subject prediction variance.The calibration set can optionally fine-tune the network using pairs of reference images and the same loss.Source: §4.1 · pp. 3–4

4.2 Differential network architecture.

  • Parallel convolutional branches extract features from both eye images, which are concatenated and passed through fully connected layers to predict their gaze difference.The architecture is described as a trade-off between prediction capacity and running time.Source: §4.2 · p. 4

4.3 Loss function, network training and adaptation

  • Training uses random same-eye image pairs and a loss on predicted versus labeled gaze differences; test-time adaptation applies the same loss to calibration pairs.Training uses Adam for 20 epochs with batch size 128; adaptation uses all possible calibration pairs for 10 epochs at a fixed learning rate of 2e-4.Source: §4.3 · p. 4

4.4 Gaze inference at test time

  • At test time, the method predicts differences between a test eye image and calibration references, then adds them to reference gazes to estimate the test gaze.Reference contributions are weighted according to the predicted difference; the paper uses a zero-mean Gaussian with σ = 0.1 radian across datasets.Source: §4.4 · p. 4

5 EXPERIMENTAL RESULTS AND ANALYSIS

The experiments compare gaze-estimation methods on three public eye-gaze datasets and examine factors including reference selection, weighting, architecture, and model complexity. The supplied passages describe datasets, evaluation protocols, and tested models but do not report comparative result values.

5.1 Datasets

  • The evaluation covers EYEDIAP, MPIIGaze, and UT-Multiview, which differ in recording conditions, image preparation, and availability of head-pose information.EYEDIAP uses continuous-screen videos; MPIIGaze includes varied head poses and illumination; UT-Multiview includes real and synthesized samples and uses head pose as input.Source: §5.1 · p. 5

5.2 Experimental protocol

  • EYEDIAP and MPIIGaze use leave-one-subject-out validation, while UT-Multiview uses three-fold validation, training on real and synthesized data but testing only on real data.The error is the average of average angular errors computed for each fold.Source: §5.2 · p. 5
  • The study evaluates average angular error in degrees for left eyes, right eyes, and their average across three public datasets.The protocol randomly selects nine reference points 200 times for linear adaptation and differential-network methods unless otherwise stated.Source: Table 1 · p. 5, §5.2 · p. 5
  • The comparisons include a generic CNN baseline, SVR and linear adaptation, and differential-network variants with and without adaptation or Gaussian averaging.Diff-VGG uses a pretrained VGG-16 backbone; the baseline does not require calibration data.Source: §5.2 · p. 5

5.3 Experimental results

The experiments find that Diff-NN outperforms the other adaptation methods on average across three datasets, including when calibration samples are scarce. Linear adaptation improves baseline results, while fine-tuning and a deeper VGG-16 backbone provide further gains at additional computational cost.

5.3 Experimental results

  • Linear adaptation decreases baseline error by 27.7% and 43.3% on EYEDIAP, 24.7% and 21.8% on UT-Multiview, and 4.9% and 9.4% on MPIIGaze, for the left and right eyes, respectively.The authors interpret the larger gains on EYEDIAP and UT-Multiview as session-based adaptation, compared with more subject-based adaptation on MPIIGaze.Source: §5.3 · p. 6
  • Diff-NN’s average error across three datasets is 17.4% lower than Lin-Ad’s and 30.6% lower than SVR-Ad’s.It improves over SVR-Ad for all datasets and users on average, and over Lin-Ad except for 2 of 50 UT-Multiview users.Source: §5.3 · p. 6
  • With 9 references sampled from one day and tests on other days, Diff-NN error rose only from 4.64° to 4.83°; removing its weighting scheme reduced performance by around 0.2°.On more session-based datasets, the reported gain was around 10% on average.Source: §5.3 · p. 6
  • Using a VGG-16 backbone reduces errors by 4%, 9%, and 18% on EYEDIAP, UT-Multiview, and MPIIGaze, respectively, at the cost of greater memory use and computational complexity.On MPIIGaze, Diff-VGG reports 3.8° error versus 4.2° for the adaptation method in.Source: §5.3 · p. 6

5.4 Cross-dataset experiments.

The cross-dataset experiment tests training on UT-Multiview and evaluation on MPIIGaze, where preprocessing differs. Adaptation improves on the baseline, but performance remains worse than within-dataset results, and the authors attribute the remaining gap partly to geometric domain shifts.

5.4 Cross-dataset experiments.

  • Cross-dataset experiments are valid for generalization when preprocessing and task formulation are equivalent, but different gaze coordinates and preprocessing can introduce systematic domain bias.The tested datasets share a similar normalization goal but not the same preprocessing.Source: §5.4 · p. 6
  • With UT-Multiview training and MPIIGaze testing, adaptation errors remain 8.1°–9.8° with references, versus within-dataset errors of 3.8°–5.6°; the authors report no added robustness to geometric domain shifts.They identify preprocessing discrepancies and differences in head- and eye-pose distributions as likely contributors.Source: §5.4 · p. 6
  • With 9 reference samples, Lin-Ad, SVR-Ad, and Diff-NN errors were 9.2°, 9.7°, and 9.8°; with 50 samples, they were 8.4°, 8.1°, and 8.4°, respectively, versus the 17.8° baseline.Prior methods reported 8.9° without any reference samples.Source: §5.4 · p. 6

5.5 Impact of reference samples

The section examines how calibration-sample selection and count affect adaptation. Diff-NN is less sensitive to calibration-point choice and works well with few references, while methods using fitted parameters benefit differently as more references become available.

5.5 Impact of reference samples

  • Diff-NN is less sensitive than Lin-Ad to random calibration-point selection, with more concentrated error distributions across trials and users.Fig. 7 compares error histograms and cumulative histograms for a given user and for all MPIIGaze users.Source: §5.5 · p. 7, Figure 7 · p. 7
  • Predictions are more accurate when test samples are closer to the reference; yaw error stays relatively flat while pitch error increases with angular separation.The authors relate the difference to stable eye-corner and iris-edge cues for yaw versus moving eyelid contours and partly hidden iris boundaries for pitch.Source: §5.5 · p. 7, Figure 8 · p. 7
  • With few references, SVR-Ad and Lin-Ad can underperform the baseline; as reference count grows, Lin-Ad error decreases significantly and SVR-Ad gradually catches up.Fig. 9 plots average angular error against the number of reference images on EYEDIAP.Source: §5.5 · p. 7, Figure 9 · p. 7
  • Diff-NN outperforms the other methods with small reference sets, even one sample, but Diff-NN-Ad better leverages larger sets: at 25 references, Diff-NN and SVR-Ad errors are 23% and 29% higher, and at 50 they are 35% and 27% higher.Diff-NN infers differences from image context without learning a model or parameter from the references; Diff-NN-Ad does not share Diff-NN’s unchanged prediction bias and variance with more data.Source: §5.5 · p. 7

5.6 Reference weighting scheme

The weighting scheme combines gaze predictions derived from multiple reference images, with a Gaussian kernel bandwidth controlling how reference contributions vary. The reported error changes little across a broad range of bandwidths.

5.6 Reference weighting scheme

  • A small σ makes mainly the closest reference contribute, whereas a large σ gives reference samples more similar contributions.The scheme's Gaussian kernel bandwidth controls the relative reference weights.Source: §5.6 · pp. 7–8
  • As σ varies from 5.7° to 22.8°, the error stays low and changes within a narrow range of 0.1°, indicating robust performance to the weighting choice.The authors report similar robustness on MPIIGaze and UT-Multiview; Fig. 10 shows average angular error on EYEDIAP by bandwidth.Source: §5.6 · p. 8, Figure 10 · p. 8

5.7 System design

The system-design comparison finds that intermediate fusion performs best among the tested architectures. The proposed intermediate representations are intended to support implicit eye registration while retaining information useful for differential gaze regression.

System design

  • Intermediate representations are intended to support implicit eye registration while retaining high-level information relevant to differential gaze regression.The authors contrast this design with early fusion of raw images and late fusion of F2 feature maps.Source: §5.7 · p. 8
  • The multi-task absolute-plus-difference Sys-5, followed by Lin-Ad, performs worse than all other systems.Sys-5 corresponds to the siamese system trained on both absolute gaze and gaze differences, then adapted.Source: §5.7 · p. 8
  • Sys-2 (intermediate F1-map fusion) achieves the best results, outperforming the early-, late-, and siamese-fusion systems.The comparison is reported for average angular error on EYEDIAP.Source: §5.7 · p. 8, Table 2 · p. 8

5.8 Algorithm complexity

The differential method requires comparisons with reference images, but reference-feature reuse and parallel mini-batch computation keep its measured running-time complexity close to the Baseline.

Algorithm complexity

  • Precomputing reference-image feature maps reduces inference to extracting the input features and computing Nc gaze differences in parallel.The authors state that this makes complexity less than Nc times that of the Baseline.Source: §5.8 · p. 8
  • Unlike Diff-NN, linear adaptation requires only the computation of Eq.(2), which has negligible computational cost.This cost is compared with the CNN Baseline.Source: §5.8 · p. 8
  • With Nc=9, measured Diff-NN complexity is close to the Baseline.Table 3 compares running times for the Baseline and Diff-NN options.Source: §5.8 · p. 8

6 CONCLUSION

The paper uses subject-specific calibration images with a differential neural network to improve appearance-based gaze estimation. Across sessions, the method produces more robust and stable results, with some additional runtime compared with a baseline CNN.

6 CONCLUSION

  • Across sessions, Diff-NN produces more robust and stable results, at some additional runtime compared with a baseline CNN.The conclusion contrasts this with linear adaptation, which can boost results in single-session situations.Source: §6 · p. 8
  • The proposed differential NN predicts gaze differences rather than directions to alleviate annoyance factors such as illumination and cropping variability.The conclusion also names variability in eye shapes as an annoyance factor.Source: §6 · p. 8
  • Linear adaptation can boost single-session results, while further network fine-tuning can leverage larger calibration sets.Source: §6 · p. 8
  • Experiments on three public, commonly used datasets support the efficacy of the proposed methods.Source: §6 · p. 8
Loading 1904.09459v3…