Source-linked AI summary
POSEidon: Face-from-Depth for Driver Pose Estimation
Guido Borghi, Marco Venturelli, Roberto Vezzani, Rita Cucchiara
TL;DR
Driver pose estimation is difficult under changing illumination, occlusions, and extreme poses. The paper introduces a depth-based framework combining localization, multimodal pose regression, and Face-from-Depth reconstruction, and reports real-time performance with strong benchmark results.
Problem
The central problem is estimating driver head and upper-body pose when illumination changes, occlusions, and extreme poses challenge conventional vision methods.
Method
POSEidon uses head localization plus three fused CNN branches processing depth, Face-from-Depth, and motion data for pose regression.
Results
Under 2° mean error is reported for all three angles on the Biwi dataset, while shoulder-pose accuracy exceeds 92%.
Takeaways & Limitations
The framework’s depth-only design supports operation in illumination-variable automotive settings, and Pandora broadens evaluation with head and shoulder annotations.
Takeaways & Limitations
ICT-3DHP has few subjects and limited frames, making it unsuitable for deep-learning approaches.
Abstract
from arXiv · showhide
Fast and accurate upper-body and head pose estimation is a key task for automatic monitoring of driver attention, a challenging context characterized by severe illumination changes, occlusions and extreme poses. In this work, we present a new deep learning framework for head localization and pose estimation on depth images. The core of the proposal is a regression neural network, called POSEidon, which is composed of three independent convolutional nets followed by a fusion layer, specially conceived for understanding the pose by depth. In addition, to recover the intrinsic value of face appearance for understanding head position and orientation, we propose a new Face-from-Depth approach for learning image faces from depth. Results in face reconstruction are qualitatively impressive. We test the proposed framework on two public datasets, namely Biwi Kinect Head Pose and ICT-3DHP, and on Pandora, a new challenging dataset mainly inspired by the automotive setup. Results show that our method overcomes all recent state-of-art works, running in real time at more than 30 frames per second.
1. Introduction
The paper targets driver head and upper-body pose estimation under illumination changes, occlusions, and extreme poses. It proposes a depth-only real-time framework, a Face-from-Depth architecture, and the Pandora dataset.
- Driver pose estimation supports monitoring attention during activities such as talking, smartphone use, falling asleep, and mirror checking.
- Depth-based sensing addresses illumination limitations that make intensity images unreliable in vehicles.
- POSEidon combines head localization with head and shoulder pose estimation from depth data, operating in real time at 30 fps.
- Face-from-Depth reconstructs gray-level face images directly from depth maps to recover appearance information useful for pose estimation.
- Pandora is introduced as a dataset containing high-resolution depth data with head and shoulder pose annotations.
2. Related Work
Prior work estimates head pose using facial features, raw appearance, 3D registration, or optimization, but complete driver-oriented upper-body frameworks remain limited. POSEidon is presented within this broader landscape of multimodal and depth-based approaches.
- Feature-based methods depend on visible facial or pose-dependent features and can fail when those features are not detected.
- Appearance-based methods use raw RGB or depth images with classifiers or neural networks for head-pose prediction.
- 3D registration methods construct or fit head models using facial point clouds, intensity, depth, or morphable models.
- Other approaches formulate head-pose estimation as optimization using PSO, ICP, least squares, or regression.
- Several methods omit head localization, while existing upper-body driver-pose work includes intrusive markers, manual initialization, or thermal imaging.
3. The POSEidon framework
POSEidon processes real-time depth frames through head localization, region-specific cropping, and separate head or shoulder pose pipelines. Its core head-pose network fuses depth, reconstructed face appearance, and motion through three CNN branches.
- A regressive CNN estimates head position and size, then supplies head or shoulder crops to the downstream pose tasks.
- Head crops feed head-pose estimation, whereas shoulder crops feed a separate shoulder-pose estimator.
- The POSEidon trident contains three CNNs processing depth, Face-from-Depth, and motion images.
- The network regresses continuous yaw, pitch, and roll Euler angles for the driver pose.
4. Face-from-Depth network
Face-from-Depth reconstructs face appearance from depth data to compensate for missing facial texture under variable illumination. Its network combines autoencoder and fully convolutional ideas in a 14-convolutional-layer architecture with a final fully connected layer.
- Motivation: Face-from-Depth estimates facial appearance from corresponding depth data, which remains available when illumination-dependent face appearance is unavailable.The approach addresses depth maps’ illumination invariance but limited texture detail.
- Architecture: The network combines autoencoder and fully convolutional neural-network design principles for face reconstruction.
- Architecture: 14 convolutional layers and one final fully connected layer define the Face-from-Depth architecture.The architecture is illustrated in Figure 4.
- Training objective: A bivariate Gaussian prior mask weights reconstruction errors spatially around the image center.The mask uses µ = [R/2, C/2]^T and empirically sets α = 3.5 and β = 2.5 for 64×64 images.
5. Pose Estimation from depth
The pose-estimation pipeline localizes the head and estimates head and shoulder orientation directly from depth. POSEidon fuses three CNN branches for continuous yaw, pitch, and roll regression, while additional cropping supports shoulder-pose estimation.
- Head localization: The head-localization network predicts the head-center coordinates under the assumption that one person occupies the foreground.Inputs are resized to 160×132 pixels, and shallow layers with small filters support real-time operation.
- Head localization: Head bounding-box width and height are computed from camera focal lengths, average face dimensions, and estimated head distance.For head pose, the average face width and height are both set to 320.
- POSEidon: POSEidon fuses three CNNs to regress continuous yaw, pitch, and roll angles.Each branch uses five convolutional layers, followed by fully connected layers with 128, 84, and 3 neurons.
- POSEidon: The fusion stage combines convolution and concatenation before three fully connected layers and two dropout regularization layers.The fusion output is processed by fully connected layers with 128, 84, and 3 activations, with dropout σ = 0.5.
- Training: POSEidon is trained in two steps: individual branch training is followed by training the final fusion layers while preserving the learned branch weights.The weighted loss gives yaw the greatest importance using weights [0.2, 0.35, 0.45].
- Shoulder pose: Shoulder pose is estimated with the head network architecture after cropping the depth image around the driver’s neck from the detected head center.The shoulder network regresses the same three pose angles.
6. Datasets
The paper evaluates Biwi Kinect Head Pose and ICT-3DHP and introduces Pandora, a public automotive-oriented dataset with head and shoulder pose annotations. Pandora combines broad pose variation, challenging occlusions, large-scale RGB/depth data, and ToF acquisition.
- Datasets: Training and testing use Biwi Kinect Head Pose, ICT-3DHP, and the newly collected Pandora dataset.Data augmentation is used to enlarge the training set and reduce overfitting.
- ICT-3DHP dataset: ICT-3DHP has about 14k frames but limited subjects and frames, making it unsuitable for deep-learning approaches.Its annotations are generated with a Polhemus Fastrack sensor mounted on a visible white cap.
- Pandora dataset: Pandora contains 110 annotated sequences recorded from a frontal fixed viewpoint simulating an in-vehicle dashboard camera.Subjects perform driving-like actions including steering, mirror checking, and gear shifting.
- Pandora dataset: Pandora provides head and shoulder pose annotations, including yaw, pitch, and roll for both body regions.Each frame also includes RGB and depth images, upper-body skeleton joints, and 2D joint coordinates.
- Pandora dataset: Subjects perform wide head and shoulder movements and use garments or objects to create realistic occlusions.Examples include glasses, scarves, caps, smartphones, tablets, and bottles.
- Pandora dataset: Pandora contains more than 250k full-resolution RGB and depth images with corresponding annotations.Depth is acquired with a Microsoft Kinect One Time-of-Flight device.
7. Experimental results
Experiments evaluate POSEidon across benchmark datasets, architectural variants, simulated occlusions, shoulder estimation, and the complete pipeline. The reported results show strong accuracy, robustness to missing data, and competitive performance across tasks.
- Ablation study: Combining all three input types produces the most significant improvement among the evaluated system configurations.The evaluated Pandora architectures include depth, Face-from-Depth, and Motion Images inputs.
- Ablation study: Depth inputs yield the lowest error rates for frontal heads, while other input types perform better for rotated poses.The component comparison also highlights averaging capabilities in POSEidon.
- Face-from-Depth evaluation: Reconstructed Face-from-Depth images produce pose-estimation results similar to those obtained with original gray-level images.This indirectly supports sufficient reconstruction accuracy for the pose-estimation task.
- Benchmark results: On Biwi, POSEidon achieves mean error below 2° for all three angles and surpasses the reported comparison methods.The authors note that some starred methods use different splits or testing procedures, making those comparisons potentially unfair.
- Occlusion robustness: Under simulated occlusions or missing data, POSEidon retains reliability on Biwi inputs.Masks are applied to remove parts of the frames and imitate realistic occlusions.
- Shoulder pose estimation: Shoulder pose estimation on Pandora reaches accuracy above 92%.The experiment compares different image crops and is conducted on Pandora because other datasets lack shoulder annotations.
- Complete pipeline: The complete pipeline provides valuable ICT-3DHP results and comparable performance with methods using depth and RGB data.The reported comparison values are 4.9±5.3, 4.4±4.6, 5.1±5.4 for pitch, roll, and yaw, respectively.
8. Conclusions and future work
The paper concludes that POSEidon is a real-time depth-only framework for head localization and driver head and shoulder pose estimation. It reports robustness to occlusions and extreme poses, while Pandora is released as a public 3D dataset.
- Conclusions: POSEidon performs head localization and driver pose estimation in real time using only depth data.The framework is intended for challenging automotive conditions involving illumination changes.
- Conclusions: The system reports results in the presence of occlusions and extreme head and shoulder poses.Its depth-only operation is described as enhancing efficacy under different illumination conditions.
- Conclusions: Pandora is proposed as a new high-quality 3D dataset and publicly released.The dataset complements the framework's evaluation and includes head and shoulder pose annotations.
- Future use: The modular architecture can omit Face-from-Depth when RGB and depth are unavailable during training, using depth plus Motion Images with lower but satisfactory performance.The complete architecture remains applicable when both modalities are available during training.