Eyemotion: Classifying facial expressions in VR using eye-tracking cameras

1. Introduction

• The method uses internally mounted infrared cameras to classify facial action units and emotive expressions without a fixed external camera.A user-facing camera may be infeasible in mobile settings and cannot capture upper-face expressions occluded by the headset.Source: §1 · p. 1
• A labeling-based approach identifies expressions and action units from limited-field-of-view eye images rather than tracking facial features.The images are also used for gaze estimation and applications such as foveated rendering.Source: §1 · p. 2
• CNNs classify select expressions from limited periocular images and outperform normal and trained human raters on the dataset.Human ratings are the evaluation baseline, not labels used during training.Source: §1 · p. 2
• The paper contributes a dataset from two commercial headsets, a personalization technique, and real-time expressive avatars for VR.Across experiments, personalization improved accuracy by 7% on average.Source: §1 · p. 2

2. Related work

• Most visual expression-classification approaches require full, unoccluded face images, which are unavailable in the headset scenario.Prior work includes facial action-unit systems, deep networks, and classifiers for expressions and emotions.Source: §2 · p. 2
• Wearable alternatives use piezoelectric or optical sensors, whereas Eyemotion uses commercial gaze-tracking cameras without customized sensors.The paper describes this as more robust to personalized fit than the cited optical-sensor approach.Source: §2 · p. 2
• VR gaze-tracking methods address gaze rather than expression or action-unit classification.The cited gaze-tracking work includes convolutional neural networks.Source: §2 · pp. 2–3
• VR facial-reenactment systems combine lower-face tracking with custom headset sensors or external cameras, unlike Eyemotion’s classification focus.Because of these optical and field-of-view constraints, the paper compares its results with eye-image CNNs and human ratings rather than reenactment systems.Source: §2 · p. 3
• Most personalization methods use per-subject samples and quick retraining, while Eyemotion’s approach requires no retraining.The paper also distinguishes its constrained expression-classification task from prior methods using unsupervised generalization with STM.Source: §2 · p. 3

3. Our Dataset

• Supervised CNN training uses expression labels collected from infrared eye images without manual image annotations.The acquisition method addresses the difficulty of labeling large amounts of data by hand.Source: §3 · p. 3

3.1. Data Collection

• The dataset covers ten upper-face action units and four basic emotive expressions, with left and right action units distinguished where applicable.Existing near-infrared eye datasets lacked expression labels and did not directly transfer to VR headset sensors; Figure 2 illustrates expression variability across conditions.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Collection Setup

• Two headsets recorded eye-image pairs at 10 Hz from 23 participants each, using different image resolutions and optical properties.Participants practiced with actor exemplars, then repeated expressions in the headset across two sessions to capture fit variation.Source: §3.2 · p. 4

3.3. Data Characteristics

• Approximately 50,000 eye-image pairs were collected per HMD (about 2,000 per participant); each expression was collected for equal time, but inability to perform every expression caused slight per-expression image-count variability.Among 46 participants, ages ranged from 18 to 64 (median 30); 11 had non-brown eyes, 4 had non-brown or black hair, and 25 were nonwhite.Source: §3.3 · p. 4

3.4. Data Cleanup

• A blink classifier and manual checks removed closed-eye images from non-eyes-closed expressions, with about 400 images removed.Neutral and closed-eye images were validated and cleaned manually.Source: §3.4 · p. 4

3.5. Data Augmentation

• Augmentation was restricted to preserve label meaning and human-like eye orientation, with 2% variation in rotation, scale, and brightness.The controlled headset environment keeps lighting and camera viewpoint largely fixed, so augmentation was carefully monitored.Source: §3.5 · p. 4

4. Our Approach

• Hand-crafted iris and eye-landmark features often failed when expressions distorted eye shape; the approach instead uses a CNN on HMD eye images.The proprietary eye landmarker was prone to failures during expressions that distorted the eyes.Source: §4 · p. 4, §4.1 · p. 4

4.1. CNN Architecture

• A pretrained Inception variant processes registered, augmented eye images that are rectified, concatenated, and scaled to 299x299 pixels.The pipeline can subtract the session’s average neutral image before the network; without personalization, it skips that subtraction.Source: §4.1 · pp. 4–5, Figure 3 · p. 5
• The model uses softmax cross-entropy with L2 regularization and RMSProp optimization.The loss compares ground-truth labels with class softmax activations across samples and classes; RMSProp uses momentum 0.9, decay 0.9, and ϵ of 1.0.Source: §4.1 · p. 5

4.2. Personalization

• Personalization averages each person/session’s first 5 seconds of neutral images, then subtracts that mean from images to form CNN test input.In practice, a new user could maintain a minimally emotive expression briefly to provide the neutral data.Source: §4.2 · pp. 5–6
• p < 0.001 for action unit expression classification and p = 0.018 for ‘emotive’ expression classification; personalization significantly increased both.The authors report that subtracting a separate mean image per user increased classifier accuracy.Source: §4.2 · p. 6

5. Experiments & Results

• The study used participant-held-out five-fold cross-validation because accuracy varied substantially across users, and benchmarked multiple CNN architectures on the expression data.The authors report roughly similar results on HMD1 and HMD2 and present HMD2 results for brevity.Source: §5 · p. 6

5.1. Human Accuracy

• The best human rater achieved F1 scores of 0.63 without and 0.65 with a neutral reference image, both below the model’s results.Inter-rater kappa was 0.61 without and 0.64 with a neutral reference image.Source: §5.1 · p. 7

5.2. Applications

• The model’s expression probabilities can drive real-time avatar expressions or adjust the VR environment, as in an example that changes ambient light and color.The authors apply exponential decay smoothing to inferred expression probabilities for temporally stable labels.Source: §5.2 · p. 7, Figure 8 · p. 8

6. Discussion

• Concatenating the two eye images outperformed alternatives by at least 10 percentage points on held-out participants, and personalization added nearly five points on average (all p < 0.05).The concatenation gain persisted when evaluation was extended across four additional participant-held-out folds.Source: §6 · p. 8
• The authors hypothesize that early eye-image concatenation lets the CNN combine information from both eyes; personalization instead uses a difference from the mean neutral image.The mean-neutral-image approach especially improved neutral classification on average across classes.Source: §6 · p. 8
• Unlike a cited approach that reached at most 48% accuracy on held-out users and required individual training, this method is described as scaling without such training.The comparison is to.Source: §6 · p. 8
• Self-reported performances can introduce incorrect labels, particularly for left and right brow raises, where many participants cannot perform the actions reliably.The authors note that autonomous collection has no accuracy check beyond participant self-report.Source: §6 · p. 8
• Instructed expressions may be biased facsimiles of real expressions, making ground-truth collection difficult; future work proposes naturalistic stimuli and expert FACS label checks.Participants can be asked to act angry but cannot be instructed to feel angry.Source: §6 · pp. 8–9
• The model assumes expressions are discrete and only one occurs at a time, so it cannot capture the breadth of facial expressiveness even with more samples.The authors propose moving beyond fixed expression sets in future work.Source: §6 · p. 9

• 74% accuracy for expression classification and 70% for facial action unit classification are achieved with the proposed personalization method without retraining.Source: §7 · p. 9
• With 50 users, the model exceeds human classification accuracy and can power an expressive surrogate avatar.Source: §7 · p. 9
• A CNN can interpret expression information in relatively low-fidelity eye images from consumer-grade equipment.Source: §7 · p. 9
• Consumer-grade eye-tracking cameras already included in VR headsets offer a means to preserve and transmit social information among users.Source: §7 · p. 9