PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaze Estimation using Transformer

Yihua Cheng, Feng Lu

arXiv:2105.14424v1cs.CV

TL;DR

Transformers had not been evaluated for gaze estimation, a task involving complex facial appearance and global relationships. The paper compares pure patch-based and CNN-transformer hybrid designs, finding that the hybrid transformer performs better with fewer parameters and reaches state-of-the-art results with pre-training.

  • Problem

    Transformer performance in gaze estimation was unexplored, despite gaze estimation’s complex facial appearance variations and transformers’ ability to capture global relations.

  • Method

    The paper compares a pure ViT-style transformer operating on image patches with a hybrid model that uses CNN features followed by a transformer encoder.

  • Results

    The hybrid transformer outperforms the pure transformer with fewer parameters and achieves state-of-the-art results on four popular gaze-estimation datasets with pre-training.

  • Takeaways & Limitations

    Hybrid transformers provide a new direction for gaze research and are more suitable than pure transformers for the evaluated gaze-estimation setting.

Abstract

from arXiv · show

Recent work has proven the effectiveness of transformers in many computer vision tasks. However, the performance of transformers in gaze estimation is still unexplored. In this paper, we employ transformers and assess their effectiveness for gaze estimation. We consider two forms of vision transformer which are pure transformers and hybrid transformers. We first follow the popular ViT and employ a pure transformer to estimate gaze from images. On the other hand, we preserve the convolutional layers and integrate CNNs as well as transformers. The transformer serves as a component to complement CNNs. We compare the performance of the two transformers in gaze estimation. The Hybrid transformer significantly outperforms the pure transformer in all evaluation datasets with less parameters. We further conduct experiments to assess the effectiveness of the hybrid transformer and explore the advantage of self-attention mechanism. Experiments show the hybrid transformer can achieve state-of-the-art performance in all benchmarks with pre-training.To facilitate further research, we release codes and models in https://github.com/yihuacheng/GazeTR.

1 Introduction

Gaze estimation requires models that handle complex, globally distributed facial appearance, while transformer performance in this task had not been established. The paper compares pure and hybrid transformers, finding the hybrid approach more effective and parameter-efficient.

  • Motivation: Facial appearance varies with head pose and illumination, making gaze estimation highly nonlinear and requiring attention to the whole appearance.
  • Motivation: Transformer performance in gaze estimation was unexplored despite transformers’ stronger ability than CNNs to capture global relations.
  • Approach: The paper evaluates pure and hybrid transformers for gaze estimation.
  • Approach: The hybrid transformer uses CNN layers to extract local feature maps before a transformer encoder captures global relations.
  • Approach: The pure transformer divides face images into patches and feeds them to a transformer encoder for gaze estimation.
  • Results: 0.18° to 0.47° improvement is achieved over ResNet-18 with only 0.2M, or 1%, additional parameters, while outperforming the pure transformer and current state-of-the-art methods with pre-training.

2 Related Work

Prior gaze-estimation work largely relies on CNN-based models, while transformers have been successful in NLP and computer vision. The paper identifies gaze estimation as an open area for transformer evaluation.

  • Gaze Estimation: CNN-based gaze methods use facial, eye, or combined face-and-eye images to improve gaze estimation accuracy.
  • Gaze Estimation: Existing CNN approaches include four-stream networks, pictorial eye representations, Bayesian inference, and coarse-to-fine face-and-eye integration.
  • Transformers: Transformers replace convolutions and recurrent networks with self-attention, layer normalization, and multilayer perceptron layers.
  • Transformers: Hybrid CNN-transformer systems have achieved better performance in computer vision tasks including object detection and instance segmentation.
  • Transformers: ViT applies a pure transformer directly to non-overlapping image patches and achieves state-of-the-art image-classification results with large-scale pre-training.
  • Research Gap: Transformer performance in gaze estimation remained unexplored, motivating this paper’s evaluation.

3 Gaze Transformers

The paper evaluates pure and hybrid transformers for gaze estimation, using either patchified images or CNN features before transformer-based gaze regression.

  • 3 Gaze Transformers: Two transformer forms are studied: a pure transformer following ViT and a hybrid transformer combining convolutional layers with a transformer.The pure model processes image patches directly, while the hybrid model preserves CNN feature extraction before capturing global relations.
  • 3.2 Pure Transformers in Gaze Estimation: The pure transformer divides face images into patches, linearly projects them into feature vectors, and adds a learnable token and positional embedding.The token aggregates patch features through self-attention and provides the representation used for gaze regression.
  • 3.2 Pure Transformers in Gaze Estimation: The pure model feeds the resulting feature matrix into a transformer and uses its first output vector with an MLP to estimate gaze.The first vector corresponds to the added token, and g denotes the estimated gaze.
  • 3.3 Hybrid Transformers in Gaze Estimation: The hybrid transformer first extracts local features with a CNN, reshapes them into a feature matrix, and then uses a transformer to capture global relations.This design addresses the difficulty of predicting gaze from local patches such as partial eye images.
  • 3.4 Implementation Details.: The implementation uses 224 × 224 × 3 face images, 2D yaw-and-pitch gaze targets, and L1 loss.GazeTR-Pure uses 14 × 14 patches, a 12-layer transformer, 64 attention heads, and hidden size 4096; GazeTR-Hybrid uses ResNet-18 features and a 6-layer transformer.

4 Experiments

Experiments pre-train the models on ETH-XGaze and evaluate them across four gaze datasets using angular error, where lower error indicates better performance.

  • 4 Experiments: ETH-XGaze pre-training uses 765K training images from 80 subjects within a dataset containing 1.1M images from 110 subjects.The images were collected with high-resolution cameras in an indoor environment, and normalized data are used directly.
  • 4 Experiments: Four datasets evaluate the models: MPIIFaceGaze, EyeDiap, Gaze360, and RT-Gene.The evaluation follows established preprocessing or protocol settings, including leave-one-person-out evaluation for MPIIFaceGaze and four-fold cross-validation for EyeDiap.
  • 4 Experiments: Training uses Adam optimization with batch size 512 and 50 epochs for both GazeTR-Pure and GazeTR-Hybrid on ETH-XGaze.The configurations include learning-rate warmup and model-specific decay steps.
  • 4 Experiments: Evaluation training uses 80 epochs, with dataset-specific batch sizes and warmup settings for MPIIFaceGaze, EyeDiap, Gaze360, and RT-Gene.GazeTR-Pure uses the same settings as GazeTR-Hybrid but changes the decay steps to 40.
  • 4 Experiments: Angular error is the evaluation metric, and a smaller error represents a better model.

4.2 Comparison with State of The Art

GazeTR-Pure is not competitive, whereas GazeTR-Hybrid achieves state-of-the-art results across all evaluated benchmarks and also outperforms AFF-Net for POG estimation.

  • GazeTR-Pure cannot achieve competitive results, while GazeTR-Hybrid shows state-of-the-art results in all benchmarks.
  • GazeTR-Hybrid is transferred to POG estimation, with gaze-direction accuracy obtained by post-processing the POG results.
  • GazeTR-Hybrid significantly outperforms AFF-Net in both POG accuracy and converted gaze-direction accuracy.

4.3 Hyper-parameters in Hybrid Transformer

The study evaluates transformer depth, attention heads, and input dimension in GazeTR-Hybrid, finding dataset-dependent effects rather than uniform gains from larger settings.

  • Three hyper-parameters are evaluated: transformer layers, multi-head self-attention heads, and transformer input dimension.
  • Layers: 36 layers outperform 6 layers in MPIIFaceGaze and RT-GENE, although performance first decreases when depth increases to 12 layers.
  • Heads: More attention heads do not consistently improve performance; one head is best on EyeDiap, while four heads are best on MPIIFaceGaze and Gaze360.
  • Input dimensions: Increasing the channel dimension to 256 improves MPIIFaceGaze and Gaze360 slightly, whereas increasing it to 512 degrades performance.

4.4 Ablation Study

Ablations indicate that self-attention and deep convolution are important components of GazeTR-Hybrid, while pre-training substantially improves transformer performance.

  • Self-attention: Removing self-attention by fixing attention weights as averages causes performance degradation, with the largest reported gap reaching 0.99° on Gaze360.
  • Pre-training: Without pre-training, GazeTR-Hybrid and GazeTR-Conv are compared across four evaluation datasets to assess pre-training and transformer effects.
  • Pre-training: Pre-training brings significant improvement, and the transformer produces a larger improvement when pre-training is used.
  • Deep convolution: Removing most convolutional layers causes obvious performance degradation, indicating that deep convolution is necessary in GazeTR-Hybrid.

4.5 Comparison with Pure CNNs.

GazeTR-Hybrid integrates CNNs with a transformer and outperforms pure-CNN comparators, including ResNet-50, while adding relatively few parameters.

  • GazeTR-Conv removes GazeTR-Hybrid’s convolutional layers and adds a two-layer MLP, producing an architecture similar to ResNet-18.
  • GazeTR-Hybrid significantly outperforms GazeTR-Conv, demonstrating the effectiveness of integrating CNNs and transformers.
  • The transformer adds only 0.2M parameters, while ResNet-50 doubles GazeTR-Conv’s parameter count.
  • GazeTR-Hybrid achieves higher performance than ResNet-50 with fewer parameters.

4.6 Impact of Pre-training

Pre-training improves both GazeTR-Hybrid and GazeTR-Conv across all evaluation datasets, with a larger impact on the transformer-based model. With pre-training, GazeTR-Hybrid outperforms GazeTR-Conv in every dataset.

  • Pre-training brings significant improvement to both GazeTR-Hybrid and GazeTR-Conv in all evaluation datasets.
  • Pre-training is more important for GazeTR-Hybrid than for GazeTR-Conv.
  • Without pre-training, GazeTR-Hybrid performs worse than GazeTR-Conv in three datasets.
  • With pre-training, GazeTR-Hybrid shows better performance than GazeTR-Conv in all datasets.

4.7 Performance Improvement of Self-attention.

The paper analyzes self-attention by comparing GazeTR-Hybrid with a version without self-attention and relating performance changes to head pose. Improvements are larger in outer head-pose regions than in central regions.

  • Self-attention is evaluated by comparing GazeTR-Hybrid with GazeTR-Hybrid without self-attention across head-pose clusters.Clusters represent 1° × 1° local yaw-and-pitch regions, and regions with performance degradation are removed.
  • Outer head-pose regions have larger performance improvements from self-attention than central regions.
  • GazeTR-Pure has 104M parameters, whereas GazeTR-Hybrid has 11.4M parameters and achieves competitive gaze-estimation performance.
  • The experiments indicate potential for transformers in gaze estimation through the effectiveness of self-attention.
  • The discussion also examines pre-training models on large-scale gaze datasets rather than relying only on ImageNet pre-training.

5 Conclusion

The paper explores pure and hybrid transformers for gaze estimation and finds that hybrid transformers perform better, achieving state-of-the-art results on four popular datasets with pre-training.

  • The paper evaluates pure and hybrid transformers for gaze estimation.
  • Hybrid transformers perform better than pure transformers in the reported experiments.
  • With pre-training, hybrid transformers achieve state-of-the-art results on four popular datasets.
  • The paper provides a new direction for future gaze research.
Loading 2105.14424v1…