PaperlayerView exact revision on arXiv

Source-linked AI summary

Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling

arXiv:1505.05916v1cs.CV

TL;DR

Eye-vision systems need large, accurately labelled datasets, but collecting and annotating them is costly and unreliable. The paper synthesizes perfectly labelled photorealistic eye images with dynamic, controllable models and varied poses, gaze directions, and illumination. SynthEyes outperforms state-of-the-art methods for eye-shape registration and cross-dataset appearance-based gaze estimation in the wild.

  • Problem

    Large-scale supervised eye-vision methods require time-consuming data collection and accurate annotations, which can be expensive, tedious, and error-prone.

  • Method

    Dynamic eye-region models built from head scans are randomly posed and rendered to synthesize realistic, perfectly labelled eye images across poses, gaze directions, and illumination conditions.

  • Results

    SynthEyes outperforms state-of-the-art methods for eye-shape registration and cross-dataset appearance-based gaze estimation in the wild.

  • Takeaways & Limitations

    Controllable synthetic eye data supports both eye-shape registration and appearance-based gaze estimation, with potential use in other realistic-eye imaging tasks.

Abstract

from arXiv · show

Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model's controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

I. INTRODUCTION

The paper addresses costly and unreliable eye-data annotation by synthesizing photorealistic, perfectly labelled training images for eye-shape registration and gaze estimation. Its dynamic, controllable models support varied appearance, pose, and illumination, enabling strong performance across both tasks.

  • Supervised eye-vision methods need large datasets, but collecting images and accurate ground-truth annotations is time-consuming, expensive, and error-prone.
  • Learning by synthesis reduces human labor and can target specific application scenarios, but realistic eye-region modeling remains difficult because eyes undergo dynamic shape changes.
  • The proposed method uses dynamic, controllable eye-region models to render large-scale realistic images for eye-shape registration and appearance-based gaze estimation.
  • The model-preparation pipeline converts dense head scans into animation-ready models, restores high-resolution skin detail, and manually annotates iris and eyelid landmarks.
  • Synthesized training data includes broad realistic appearance variation through image-based lighting and is reported to outperform state-of-the-art methods on both target tasks.

II. RELATED WORK

Prior synthetic-data methods show promise, but eye applications require training distributions and rendering realism that simpler models may not provide. This work emphasizes controllable, realistic synthesis with labels usable beyond gaze estimation.

  • Learning-based performance depends on how well training data covers the test distribution, motivating synthetic data when comprehensive real-data collection is difficult.
  • Realism is especially important for complex objects such as eyes, and illumination differences can substantially affect gaze-estimation accuracy.
  • Earlier eye-synthesis work used rigid, low-resolution models with gaze labels but lacked a parametric model and was not easily transferable to different tasks.
  • The proposed realistic, fully controllable model synthesizes close-up eye images with ground-truth landmark positions, enabling learning-by-synthesis for eye-shape registration.

B. Computational Modeling of the Eyes

The paper builds a dynamic eye-region model from scanned head geometry and a posable eyeball model, targeting realistic variation in facial appearance, eye shape, motion, and rendering.

  • The model combines 3D head-scan geometry with a posable eyeball model to generate fully labelled training images through random posing.
  • The head-model collection varies eye shape, surrounding bone structure, skin smoothness, and skin color across close-up eye regions.
  • The eyeball model represents the sclera, pupil, iris, and cornea while supporting pupillary-dilation shape variation and iris-color and scleral-vein texture variation.
  • Dynamic modeling aims to represent continuous appearance changes during eye movement, requiring deformable geometry that is correctly topologized and rigged.

A. Simplified Eyeball Model

The simplified eyeball model uses joined transparent and refractive geometry plus an inner iris-and-sclera component, with controllable shape and texture variation for realistic rendering.

  • The eyeball comprises an outer joined two-sphere geometry modeling the eye and corneal bulge, plus an inner flattened sphere representing the iris, pupil, and sclera.The outer component is transparent, refractive, and partially reflective, while a 0.5mm gap represents corneal thickness.
  • Shape variation is modeled with blend shapes for dilated and constricted pupils and approximately 10% variation in iris size.
  • Texture variation uses separate sclera tint, iris photo-texture, and veins layers, including three sclera colors, four iris textures, and two vein appearances.
  • Realistic eye-region rendering also depends on high-quality scanned facial detail because eye shape, orbital structure, and skin detail vary substantially across people.

C. Eye-Region Geometry Preparation

The eye-region geometry is retopologized and augmented with dynamic components so eyelids and eyelashes deform realistically with gaze.

  • C. Eye-Region Geometry Preparation: The geometry preparation addresses the need for controllable, low-resolution topology rather than directly using high-resolution scan geometry.The original topology is difficult to control during eye-region shape changes.
  • C. Eye-Region Geometry Preparation: Retopologized edge loops follow exterior eye muscles, enabling realistic eye-region deformations.The low-poly mesh sacrifices scan detail, which is restored with a displacement map.
  • C. Eye-Region Geometry Preparation: Eyelid blend shapes interpolate with eyeball pitch, producing continuous facial deformation across gaze directions.This replaces discrete gaze-specific head scans with densely varying facial poses.
  • C. Eye-Region Geometry Preparation: Eyelashes are simulated as directed hair particles because they can occlude the eye and affect eye-tracking algorithms.The particles are generated from a manually placed control surface and curled during growth.

IV. TRAINING DATA SYNTHESIS

Training images are synthesized by varying camera viewpoint, gaze, lighting, and eye-model configuration to represent real-world appearance variation.

  • IV. TRAINING DATA SYNTHESIS: The synthesis process aims to sample continuous viewpoint and appearance variation representative of in-the-wild eye images.The section covers viewpoint and model posing, image-based lighting, landmark annotation, and rendering.
  • IV. TRAINING DATA SYNTHESIS: Camera positions are sampled over spherical coordinates around the eyeball center to simulate head-pose changes.Orthographic rendering approximates an eye region cropped from a wide-angle camera image.
  • IV. TRAINING DATA SYNTHESIS: At each camera position, multiple gaze vectors generate images of the eye looking in different directions.The rendered image is determined by camera position, gaze vector, lighting environment, and eye-model configuration.
  • IV. TRAINING DATA SYNTHESIS: Rendering is restricted to valid eyeball rotations and pupil positions that keep sufficient iris visible within the eyelid landmarks.The constraints are |α|≤25° and |β|≤35°.

B. Creating Realistic Illumination

SynthesEyes models realistic illumination with randomly selected, rotated, and intensity-scaled HDR environment maps, while producing automatically deformable landmark annotations and photorealistic renders.

  • B. Creating Realistic Illumination: Image-based lighting uses HDR panoramic environments to provide realistic illumination across varied eye appearances.Environment images are projected around the object and sampled as light intensity during rendering.
  • B. Creating Realistic Illumination: Four freely available HDR environments are randomly selected, rotated, and intensity-scaled to vary lighting conditions.Rotation simulates head-pose-related environmental changes, while intensity scaling simulates ambient-light changes.
  • B. Creating Realistic Illumination: Each 3D eye-region model carries 28 landmarks for eyelids, iris boundaries, and pupil boundaries.Iris and pupil landmarks are tied to eyeball vertices so they deform automatically with pupil and iris size.
  • B. Creating Realistic Illumination: Cycles path tracing renders physically based materials with 150 rays per pixel to produce noise-free, photorealistic images.The generic dataset contains 11,382 images covering 40° of viewpoint variation and 90° of gaze variation.

V. EXPERIMENTS

The synthesized data is evaluated on eye-shape registration and appearance-based gaze estimation, addressing landmark detection and person-independent gaze prediction under varying appearance.

  • V. EXPERIMENTS: The method is evaluated on eye-shape registration and appearance-based gaze estimation.These tasks test the usefulness of the synthetic data generation method.
  • V. EXPERIMENTS: Eye-shape registration detects eyelid, iris, and pupil landmarks using image features or statistically learned deformable models.The synthesized dataset provides automatically labelled training data with consistent labels across viewpoints and people.
  • V. EXPERIMENTS: Person-independent gaze estimation requires training data spanning eye shapes, head poses, gaze directions, and illumination conditions.The training data must cover appearance changes without relying on data from the target user.

A. Eye-Shape Registration

SynthesEyes supports accurate eye-shape registration in unconstrained images, matching or outperforming comparison systems while showing that realistic lighting and eyelid motion matter.

  • A. Eye-Shape Registration: The model was evaluated on 830 visible-eye images from the 300-W validation datasets and on 188 annotated MPIIGaze images.The 300-W experiment targeted eyelid boundaries, while MPIIGaze supplied eyelid and iris boundaries.
  • A. Eye-Shape Registration: CLNF Synth matched CLNF 300-W on eyelid localization, with both achieving Mdn = 0.0110 using different training data.CLNF Synth used only 10 participants in four lighting conditions.
  • A. Eye-Shape Registration: The eye-specific CLNF outperformed SDM, face-CLNF, DRMF, and Tree based systems for eyelid localization.Reported median errors were 0.0134, 0.0139, 0.0238, and 0.0217, respectively.
  • A. Eye-Shape Registration: Removing eyelid motion or realistic lighting worsened registration, producing Mdn = 0.0129 and Mdn = 0.0120, respectively.These ablations lacked important degrees of variability in the training sets.
  • A. Eye-Shape Registration: On webcam images, CLNF Synth achieved comparable iris-fitting accuracy to EyeTab, with Mdn = 1.48px versus Mdn = 1.44px.EyeTab failed to terminate in 2% of test cases, whereas CLNF Synth was described as more robust.

B. Appearance-Based Gaze Estimation

SynthesEyes enables cross-dataset appearance-based gaze estimation and benefits from matching synthesized pose and gaze distributions to the target setting, especially when combined with real-data fine-tuning.

  • B. Appearance-Based Gaze Estimation: A CNN trained on generic SynthesEyes achieved similar MPIIGaze cross-dataset performance to one trained on UT, with µ = 13.91° versus µ = 13.55°.The evaluation used the same CNN architecture and tested both synthetic datasets on MPIIGaze.
  • B. Appearance-Based Gaze Estimation: The synthetic, MPIIGaze, and UT datasets differ in their gaze-direction and head-pose distributions.These distributions are shown in Figure 11.
  • B. Appearance-Based Gaze Estimation: Combining synthetic datasets improved MPIIGaze performance relative to individual training sets, while some eye models generalized better than others.Per-eye-model results identified models such as f3, m2, and m4 as having poor generalization.
  • B. Appearance-Based Gaze Estimation: Matching the training data’s head-pose and gaze ranges to MPIIGaze improved performance, and SynthesEyes exceeded UT by 0.74° with p<0.0001.The targeted SynthesEyes setting used 10° pose and 20° gaze variation.
  • B. Appearance-Based Gaze Estimation: Fine-tuning SynthesEyes-trained models on UT improved results in both settings, reaching µ=11.12° untargeted and µ=7.90° targeted.The untargeted result significantly outperformed the state-of-the-art result with p<0.0001.

VI. CONCLUSION

The paper presents SynthesEyes, a graphics pipeline for realistic, perfectly labelled eye images across pose, gaze, and illumination variation. It reports strong performance for eye-shape registration and cross-dataset gaze estimation.

  • VI. CONCLUSION: SynthesEyes uses dynamic eye-region models from head scans to generate realistic, perfectly labelled close-up eye images across head poses, gaze directions, and illumination conditions.The pipeline is based on computer graphics and controllable model variation.
  • VI. CONCLUSION: The synthesized data outperformed state-of-the-art methods for eye-shape registration and cross-dataset appearance-based gaze estimation in the wild.The conclusion identifies both tasks as demonstrations of SynthesEyes’ usefulness.
Loading 1505.05916v1…