PaperlayerView exact revision on arXiv

Source-linked AI summary

Traffic-Context-Aware Driver Emotion Recognition with Expectation-Deviation-Driven Inference

Longhao Yan, Jinhao Liang, Kaidi Yang

arXiv:2610.33713v1cs.CV

TL;DR

Driver emotion recognition matters for understanding how drivers perceive traffic, yet existing approaches largely rely on in-cabin signals and overlook dynamic surrounding-vehicle interactions. TADER combines facial-image recognition with trajectory-based expectation deviations and attention-based fusion, using transfer learning to address limited paired data. It outperforms state-of-the-art methods in the reported naturalistic and simulated settings, with accuracy improvements on AIDE and manD.

  • Problem

    Many driver emotion-recognition approaches rely primarily on in-cabin signals, while dynamic interactions among surrounding vehicles are rarely modeled despite their relevance to driver emotion.

    Source: §I · p. 1

  • Method

    TADER combines a vision-based in-cabin recognizer with traffic-context features based on predicted-versus-observed vehicle trajectories, fused through attention, and transfers trajectory-prediction knowledge from large-scale public data.

    Source: §I · p. 2, A. Vision-Based Base Recognizer · p. 3, 2) Attention-Based Embedding Fusion · p. 5

  • Results

    4.82% accuracy improvement on AIDE and 5.29% on manD; TADER consistently outperformed state-of-the-art methods across naturalistic and simulated driving environments.

    Source: §V · p. 12

  • Takeaways & Limitations

    On AIDE, traffic context provides complementary information that helps resolve visually ambiguous emotion cases, although it is not sufficient for fine-grained recognition by itself.

    Source: C. Performance Analysis · p. 8

Abstract

from arXiv · show

Driver emotion provides an important indication of how drivers perceive and evaluate ongoing traffic situations, making reliable emotion recognition an important capability for effective human-vehicle interaction. Yet, most existing driver emotion recognition approaches rely primarily on in-cabin images and overlook how surrounding traffic interactions shape driver emotional responses. To address this limitation, we propose a traffic-context-aware driver emotion recognition (TADER) framework that integrates in-cabin signals with out-of-cabin traffic scenarios, modeling driver emotions as arising from expectation deviation represented by the discrepancy between predicted and observed vehicle trajectories. The proposed framework comprises two key components: (1) a vision-based base recognizer that extracts spatiotemporal representations from in-cabin facial images to infer driver emotions, and (2) a traffic-context-aware fusion module that quantifies expectation deviation from external traffic scenarios and integrates it into the base recognizer via attention-based embedding fusion. To overcome the scarcity of paired in-cabin and out-of-cabin data, we introduce a transfer learning strategy that transfers a trajectory prediction model trained on large-scale naturalistic datasets to smaller paired datasets. Extensive experiments demonstrate that TADER consistently outperforms state-of-the-art methods across both naturalistic and simulated driving environments by incorporating traffic context information into emotion recognition, resulting in improvements of 5% in accuracy on the AIDE and manD datasets.

I. INTRODUCTION

Driver emotion recognition matters for human-vehicle interaction, but existing approaches largely rely on in-cabin signals and often omit dynamic traffic interactions. TADER combines cabin observations with trajectory-based expectation deviations, while the problem formulation and method address recognizing emotion from synchronized visual and traffic data.

I. INTRODUCTION

  • Prior emotion-recognition approaches mainly model in-cabin signals, while the one cited traffic-aware method considers preceding-vehicle distance but not dynamic, time-evolving interactions.In-cabin facial analysis can also be sensitive to occlusion, lighting variation, and individual differences.Source: §I · p. 1
  • TADER combines in-cabin visual signals with out-of-cabin traffic context, representing expectation deviation through mismatches between predicted and observed vehicle trajectories.The framework also uses attention-based embedding fusion and transfers a trajectory predictor trained on large-scale public driving data to paired datasets.Source: §I · p. 2

II. PROBLEM FORMULATION

  • The task is to recognize each interval’s emotion from synchronized in-cabin observations, historical vehicle trajectories, and realized trajectories over a prediction horizon.The scene represents the ego vehicle and observable surrounding vehicles as a graph, with edges indicating interactions such as same-lane succession or adjacent-lane positions.Source: §II · p. 2

III. METHODOLOGY

  • The method pairs a facial-image spatiotemporal recognizer with a traffic-context module that quantifies expectation deviation and fuses it through attention-based embeddings.A transfer-learning strategy trains a trajectory predictor on large-scale naturalistic data before transferring its representations to a smaller paired dataset.Source: §III · p. 2

A. Vision-Based Base Recognizer

The base recognizer infers driver emotion from in-cabin camera data using CAERNet, which jointly models facial expressions and background information. Its dual-stream features are attended to and fused into a visual embedding for emotion prediction.

A. Vision-Based Base Recognizer

  • CAERNet jointly models facial expressions and background information in in-cabin images, and the framework can also use other vision-based emotion-recognition architectures.The authors select CAERNet for its empirical performance.Source: A. Vision-Based Base Recognizer · p. 3
  • CAERNet separates each camera interval into a facial stream for expression recognition and a background stream carrying auxiliary cues such as posture and gesture.Its dual-branch 3D CNN encoder extracts spatiotemporal representations from the streams.Source: A. Vision-Based Base Recognizer · p. 3
  • Attention emphasizes informative spatial regions in the background stream before its refined features are fused with facial features using learnable weights.The resulting visual feature embedding is processed to predict the emotion distribution.Source: A. Vision-Based Base Recognizer · pp. 3–4
  • The vision-based recognizer is trained with cross-entropy loss against the ground-truth emotion label.The loss is defined for the predicted visual emotion distribution.Source: A. Vision-Based Base Recognizer · p. 4

B. Traffic-Context-Aware Fusion Module

The fusion module characterizes traffic context through discrepancies between predicted and observed multi-vehicle dynamics. It encodes those deviations and vehicle information as a context embedding for emotion recognition.

1) Expectation-Deviation-Based traffic context Feature Extraction:

  • The module predicts interaction-aware trajectories from historical traffic observations, then compares predicted and observed trajectories to quantify expectation deviation.The predictor is based on Pishgu but can use any architecture that models surrounding dynamics as interaction graphs.Source: 1) Expectation-Deviation-Based traffic… · p. 4
  • A graph interaction representation is concatenated with historical trajectories and refined by convolutional blocks with channel and spatial attention.The hierarchical processing extracts spatiotemporal dependencies from the fused representation.Source: 1) Expectation-Deviation-Based traffic… · p. 4
  • Vehicle-specific Gaussian distributions represent predicted future trajectories, while each vehicle’s deviation and covariance-normalized agreement define its expectation deviation and prediction confidence.These quantities are organized according to the vehicle-node order in the interaction graph.Source: 1) Expectation-Deviation-Based traffic… · p. 4
  • The context encoder applies self-attention to deviation sequences and transforms actual trajectories and prediction confidence before concatenating the representations into a traffic-context embedding.The embedding is used to compute an emotion distribution.Source: 1) Expectation-Deviation-Based traffic… · p. 4

2) Attention-Based Embedding Fusion:

The fusion component combines visual and traffic-context embeddings through residual multi-head cross-attention. It also jointly supervises visual, context, and fused emotion predictions.

2) Attention-Based Embedding Fusion:

  • The visual embedding supplies the query, while the traffic-context embedding supplies the key and value in residual multi-head cross-attention.An unmasked zero key–value pair lets the visual query regulate the traffic-context contribution.Source: 2) Attention-Based Embedding Fusion · p. 5
  • Attention-weighted traffic-context information joins the visual embedding through a residual connection to form the fused representation for emotion prediction.Both modalities contribute to the final emotion prediction.Source: 2) Attention-Based Embedding Fusion · p. 5
  • Training jointly supervises visual, traffic-context, and fused emotion predictions, with separate weights balancing their loss contributions.The fusion-based cross-entropy loss aligns the fused prediction with the ground truth.Source: 2) Attention-Based Embedding Fusion · p. 5

C. Transfer Learning for Limited Paired Data

Because paired in-cabin and traffic observations are costly to collect at scale, the method transfers trajectory-prediction representations from large naturalistic datasets to limited paired data. The target predictor is fitted to target trajectories and aligned with a fixed source predictor.

C. Transfer Learning for Limited Paired Data

  • Paired in-cabin driver-state and out-of-cabin traffic data are costly to collect at scale, motivating transfer from large naturalistic driving datasets.The paper cites regularities in traffic interaction patterns as supporting generalization across contexts, drivers, or datasets.Source: C. Transfer Learning for Limited… · p. 5
  • Source- and target-domain trajectory predictors are initialized from source pretraining; the source model stays fixed while the target model updates on limited paired data.The source model captures generalizable driving behaviors.Source: C. Transfer Learning for Limited… · p. 5
  • Target training combines Gaussian negative log-likelihood fitting to target trajectories with KL-divergence alignment between source- and target-model predicted distributions.The overall training objective also balances the trajectory-prediction loss with the other module losses.Source: C. Transfer Learning for Limited… · p. 5

IV. RESULTS

The experiments evaluate TADER on two multimodal driver emotion datasets. The section outlines performance analyses, ablations, and sensitivity analyses.

IV. RESULTS

  • The evaluation covers two multimodal driver emotion datasets.The supplied passage does not report dataset names or numerical results.Source: §IV · p. 5
  • The section reports emotion-recognition performance and analyzes contributions from the visual and traffic-context branches.Benchmark methods and experimental setup are addressed in separate subsections.Source: §IV · p. 5
  • The planned analyses also include ablation studies and sensitivity analyses.Source: §IV · p. 5

A. Benchmarks

The benchmarks span visual-only facial-emotion methods and TADER, which adds multi-vehicle traffic context to a visual branch. Their comparison is designed to test whether traffic context contributes beyond visual modeling.

A. Benchmarks

  • MTCNN-D is a CNN–LSTM facial-emotion baseline that models dynamics across sequential facial frames without traffic context.Source: A. Benchmarks · pp. 5–6
  • MAE represents self-supervised facial-expression recognition with large-scale pretraining, while IAL uses an intensity-aware loss in a transformer-based facial-only method.Source: A. Benchmarks · p. 6
  • CAERNet uses facial expressions and image background information, including driver posture, rather than relying solely on facial features.Source: A. Benchmarks · p. 6
  • TADER combines an in-cabin visual branch with multi-vehicle traffic context, while matched CAERNet comparisons isolate the traffic branch from visual-backbone capacity.The visual branch uses MTCNN-D, MAE, IAL, or CAERNet in benchmark comparisons; CAERNet is the default backbone for later analyses.Source: A. Benchmarks · p. 6

B. Experiment Setup and Implementation Details

Experiments evaluate TADER on AIDE naturalistic driving data and manD simulator data, using aligned three-second samples and a transferred trajectory predictor. Both datasets are divided into training, validation, and test splits and class-balanced.

1) Datasets:

  • AIDE captures naturalistic on-road driving, while manD provides controlled simulator observations with synchronized facial video and vehicle or environment information.Source: 1) Datasets · p. 6, Figure 2 · p. 6
  • AIDE samples are three-second clips with synchronized multi-view recordings; the included scenes are lane-changing and forward-driving, with five emotion categories.Vehicle trajectories are derived from YOLO-based detection and multi-object tracking and projected into bird’s-eye view.Source: 1) Datasets · pp. 6–7
  • manD uses controlled simulator scenarios labeled with five emotional states; both datasets are aligned as three-second clips at 15 Hz with 10 observed and 35 future trajectory steps.Each dataset is split 70%/15%/15% for training, validation, and testing, then class-balanced separately.Source: 1) Datasets · p. 7

2) Implementation Details:

  • Benchmark methods use their original default hyperparameters with minor adaptations for the datasets’ five emotion categories.Models are optimized for 50 epochs using Adam, a learning rate of 10^-4, weight decay of 10^-4, and batch size 40.Source: 2) Implementation Details · p. 7
  • The trajectory predictor is pretrained on NGSIM and adapted to AIDE and manD, with highD used as an alternative source in source-domain sensitivity analysis.The visual branch processes 16 frames; the interaction graph represents at most 12 vehicles.Source: 2) Implementation Details · p. 7

C. Performance Analysis

TADER improves performance over matched visual-only baselines on both datasets, with gains in fine-grained and coarse-grained recognition. Branch analyses indicate that traffic context provides complementary information, especially for visually ambiguous or category-specific cases.

C. Performance Analysis

  • CAERNet has the highest fine-grained Macro-F1 among visual-only baselines, scoring 88.93% on AIDE and 90.01% on manD.MTCNN-D has the lowest reported Macro-F1: 67.26% on AIDE and 53.42% on manD.Source: C. Performance Analysis · p. 7
  • On AIDE, Anxiety recall rises from 83.42% to 94.58% over CAERNet, while manD shows particularly large gains for Anger and No Emotion.The class-wise contribution varies by emotion category and dataset.Source: C. Performance Analysis · p. 7
  • Macro-F1 rises from 88.93% to 93.75% on AIDE and from 90.01% to 95.30% on manD for TADER versus matched CAERNet.TADER also improves fine-grained and coarse-grained Macro-F1 over the visual-only baseline for all four evaluated visual backbones.Source: C. Performance Analysis · p. 7, Table I · p. 8
  • On AIDE, fusion raises Anxiety recall from 87.2% in the visual branch to 94.6%, while the Traffic-only representation is substantially weaker overall.Traffic context alone is not sufficient for fine-grained emotion recognition, but provides complementary information for visually ambiguous cases.Source: C. Performance Analysis · p. 8
  • On manD, fusion raises Anger recall from 87.8% to 95.3% and No Emotion recall from 92.5% to 95.9%; fused outputs are more compact and separated.The t-SNE description reports greater overlap for traffic-context outputs, particularly on manD, than for visual outputs.Source: C. Performance Analysis · p. 8

D. Ablation Study

Ablations examine the trajectory, expectation-deviation, and prediction-confidence components, the scope of vehicle context, and trajectory-predictor transfer learning. Results favor retaining all three context components and jointly representing ego and surrounding vehicles.

D. Ablation Study

  • Removing σ causes the largest Traffic-only Fine-grained Macro-F1 drop on both datasets, followed by d, while removing ρ has a smaller effect.On AIDE, scores fall from 60.37% to 52.39% without σ; on manD, they fall from 72.22% to 57.33%.Source: D. Ablation Study · p. 8, Table II · p. 10
  • In the complete TADER model, removing ρ, d, or σ lowers Macro-F1 from 93.75% to 93.66%, 92.26%, or 91.67% on AIDE, and from 95.30% to 94.71%, 94.44%, or 93.75% on manD.The visual representation partially compensates for missing traffic-context components; σ is realized motion, d expectation-relative information, and ρ covariance-normalized prediction confidence.Source: D. Ablation Study · p. 10
  • The full multi-vehicle representation achieves higher Macro-F1 than ego-only or surrounding-only variants on both datasets, although the better restricted variant reverses between AIDE and manD.The joint representation consistently has the highest aggregate performance.Source: D. Ablation Study · p. 10
  • Transfer learning increases Fine-grained Macro-F1 from 92.20% to 93.75% on AIDE and from 94.16% to 95.30% on manD.Coarse-grained Macro-F1 also increases, from 92.64% to 94.34% on AIDE and from 94.91% to 97.28% on manD; class-wise effects remain emotion-dependent.Source: D. Ablation Study · p. 10

E. Sensitivity Analysis

TADER outperforms state-of-the-art methods on AIDE and manD, while sensitivity analyses find strong performance across trajectory-pretraining sources and predictor architectures. The conclusion also identifies expanded traffic context and personalized control as future directions.

E. Sensitivity Analysis

  • Pretraining-source changes redistribute class-wise recalls: on AIDE, highD improves Anger by 4.22 points but reduces Anxiety by 1.93 points relative to NGSIM.The passage also reports varying class-wise shifts on manD, with the largest gains for Surprise and No Emotion and slight decreases for Anger and Fear.Source: E. Sensitivity Analysis · p. 11
  • Replacing Pishgu with Trajectron++ increases fine-grained Macro-F1 from 93.75% to 94.63% on AIDE and from 95.30% to 97.49% on manD.The authors conclude TADER is not tied to one predictor architecture and can benefit from stronger trajectory prediction models.Source: E. Sensitivity Analysis · p. 11, Table VI · p. 12

V. CONCLUSION

  • The framework combines in-cabin facial representations with multi-vehicle traffic context, while transfer learning carries trajectory knowledge from large-scale public data to paired emotion-recognition datasets.Its traffic context jointly represents realized trajectories, expectation deviations, and prediction confidence.Source: §V · pp. 11–12
  • Accuracy improves by 4.82% on AIDE and 5.29% on manD, and TADER consistently outperforms state-of-the-art methods across naturalistic and simulated driving environments.The conclusion attributes these results to integrating in-cabin visual signals with out-of-cabin multi-vehicle traffic information.Source: §V · p. 12
  • Future work plans to add pedestrian behavior, road topology, and weather, apply causal inference to contextual factors, and develop continual personalized vehicle control.The proposed control strategies would use real-time emotion recognition to adapt vehicle behavior to drivers’ affective states and preferences.Source: §V · p. 12
Loading 2610.33713v1…