Source-linked AI summary
DeepBreath: Deep Learning of Breathing Patterns for Automatic Stress Recognition using Low-Cost Thermal Imaging in Unconstrained Settings
Youngjun Cho, Nadia Bianchi-Berthouze, Simon J. Julier
TL;DR
DeepBreath addresses automatic recognition of mental overload from breathing patterns without relying on handcrafted features. It turns thermal nostril measurements into respiration variability spectrograms, augments them for CNN training, and reports 84.59% binary and 56.52% multiclass accuracy. Its evaluation covers two cognitive tasks, and the authors identify ground-truth refinement and inter-person variability as improvement areas.
Problem
Prior breathing-based stress work leaves the value of breathing features unclear, while handcrafted feature extraction is the approach DeepBreath seeks to avoid.
Source: §1 · p. 1Method
DeepBreath converts thermal nostril measurements into respiration variability spectrograms, applies a CNN without handcrafted features, and augments the data with sliding crops.
Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 3Results
84.59% binary and 56.52% multiclass accuracy; the CNN exceeded a single-layer NN’s 77.31% and 53.65%, respectively, but differences were not significant.
Source: §7 · p. 7Takeaways & Limitations
The study reports that breathing signals alone can support above-chance three-class stress recognition, and concludes that stress detection performed well under unconstrained settings.
Source: §7 · p. 7, §8 · p. 7The authors identify relatively few participants as a possible reason the deep and shallow accuracy differences were not significant.
Source: §7 · p. 7
Abstract
from arXiv · showhide
We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating hand-crafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.
1. Introduction
DeepBreath aims to recognize mental overload from breathing patterns measured with low-cost thermal imaging. It represents breathing variability as spectrogram sequences, learns features with a CNN, and uses data augmentation for small datasets.
1. Introduction
- The paper proposes automatic stress recognition from breathing patterns captured with a low-cost thermal camera in unconstrained settings.The study treats stress as mental overload.Source: §1 · p. 1
- Bi-dimensional spectrogram variability sequences represent breathing-pattern changes over time, addressing prior reliance on gross statistics such as average breathing rate.The authors note that breathing irregularity can also occur in stressful situations.Source: §1 · p. 1
- A deep-learning framework learns features from spectrogram sequences without handcrafted features, while a unidirectional sliding cropper augments data for small training sets.The approach addresses the considerable training-data requirement of deep learning.Source: §1 · p. 1
- The paper investigates low-cost thermal imaging as a portable stress sensor, avoiding the wearable sensors and RGB cameras commonly used for breathing tracking.The authors note that RGB cameras raise privacy and lighting challenges.Source: §1 · p. 1
2. Background
Prior work leaves breathing’s contribution to stress recognition unclear, while thermal affect studies often used cumbersome systems and constrained tracking. DeepBreath builds on portable tracking and proposes a CNN-compatible representation for small physiological datasets.
2.1. Breathing and Stress
- Breathing-rate changes during stress are inconsistent across studies, and breathing features’ contribution is unclear relative to other physiological measures.One cited study found HRV most informative and breathing rate a minor feature.Source: §2.1 · p. 2
2.2. Thermal imaging and affect detection
- Earlier thermal-affect studies used heavy, expensive systems and limited ROI tracking, restricting movement and requiring stable-temperature environments.A prior ROI-tracking example used a dot stick.Source: §2.2 · p. 2
- A previously proposed ROI tracker worked with low-cost portable thermal cameras and was reported robust in unconstrained settings, motivating its use for stress detection during sedentary mental tasks.The reported tracking performance was comparable to a standard breathing belt.Source: §2.2 · p. 2
2.3. Deep learning for affective computing
- Earlier deep-learning affect models include facial-expression and speech-stress recognition, with reported classification performance exceeding shallow approaches.Examples include Liu et al. and Lane et al..Source: §2.3 · p. 2
- CNN methods have shown promise in affect recognition, but small datasets and one-dimensional physiological signals limit their straightforward use.The paper responds with a two-dimensional breathing representation and data augmentation.Source: §2.3 · p. 2
3. DeepBreath: Design and Algorithm
DeepBreath converts thermal nostril measurements into respiration variability spectrograms, augments those images with sliding crops, and classifies them with a compact CNN. The design targets breathing dynamics and small-data learning with a low-computational-cost network.
3.1. Respiration Variability Spectrogram (RVS) collection from thermal imaging
- Thermal videos from a portable camera positioned near the nostrils are processed with an ROI tracker to recover one-dimensional breathing signals.The camera is attached to a smartphone and placed at up to 55 cm from the nostril area.Source: §3.1 · p. 3
- The RVS is a two-dimensional spectrogram formed by stacking short-window PSD vectors over time to condense breathing variability into an image.The PSD captures short-time autocorrelation and similarities between neighboring signal patterns.Source: §3.1 · p. 3
3.2. Data augmentation
- A unidirectional sliding cropper augments the RVS dataset with square-window crops while preserving each label.The cropper addresses the narrow frequency bandwidth of breathing signals, which requires longer recordings for deep learning; Figure 2 depicts a 120x120 window and a frequency-magnitude color scale.Source: §3.2 · p. 3, §3.1 · p. 3
3.3. CNN architecture
- The CNN uses two convolutional and two pooling layers followed by a fully connected layer, targeting low computational cost for embedded or mobile use.Input patches are resized from 120x120 to 28x28, and SGD is used for supervised fine-tuning.Source: §3.3 · p. 4
4. Study Design and Dataset
The study recorded breathing during easy and difficult Stroop and Math tasks, using a low-cost thermal camera and self-reported stress ratings. The resulting data included augmented respiration spectrograms labeled by stress level.
4.1. Stress induction tasks
- Participants completed easy and difficult Stroop and Math tasks designed to elicit different cognitive loads.Stroop difficulty varied word–font colour congruence; Math difficulty varied subtraction by one versus a two-digit number.Source: §4.1 · p. 4
4.2. Participants and procedure
- After a five-minute relaxation period, participants completed both task types and reported stress on a 10-cm VAS before and after each task.Easy and difficult sessions were counterbalanced; recording used a low-cost thermal camera connected to a smartphone.Source: §4.2 · p. 5
4. Study Design and Dataset
- The eight participants’ thermal-video sessions were segmented by task and difficulty, producing 3,936 augmented 120 × 120 spectrogram images (average 492, SD=91.49, per participant), which were publicly available.The study procedure was approved by the UCL Interaction Centre Ethics Committee.Source: §4 · p. 4, §4.2 · p. 5
4.2. Participants and procedure
- Normalised self-reports were clustered into stress groups, and example spectrograms were shown for no-, low-, and high-stress ratings.The figure caption distinguishes easy and difficult Stroop and Math sessions by colour and shape.Source: §4.2 · p. 5
5. Ground truth and stress classification
The study converted participants’ self-reported stress ratings into labels for breathing spectrograms and evaluated classification using leave-one-subject-out cross-validation. It also compared the CNN with single-layer neural networks.
Ground-truth construction
- Each participant’s ratings were min–max normalised and clustered with k-means into none, low, and high stress labels for the spectrograms.The paper reports significant differences between each pair of clusters (all t-test comparisons p<0.001).Source: §5 · pp. 5–6
- The analysis excluded the initial relaxation period because its ratings did not differ significantly from the no-stress group and its lack of task-related speech and movement could simplify recognition.The relaxation-versus-no-stress comparison was t(16)=1.211, p=0.244.Source: §5 · p. 6
Stress classification
- Because low- and high-stress spectrograms shared similarities, the study also combined them into a binary stress class against no stress.The paper notes that breathing dynamics vary between individuals and appear more irregular at higher self-reported stress levels.Source: §5 · p. 6
Evaluation protocol
- The CNN and three single-layer neural networks were evaluated with eight-fold leave-one-subject-out cross-validation, training on seven participants and testing on the held-out participant.The shallow networks used RVS image patches at 28 × 28, 60 × 60, or 120 × 120 resolution.Source: §5 · p. 6
6. Results
The CNN achieved the highest reported mean accuracy in both binary and three-class stress recognition, compared with the single-layer neural networks. Evaluation used leave-one-subject-out cross-validation on augmented breathing spectrograms.
Evaluation
- The evaluation used 3,936 augmented breathing spectrograms and leave-one-subject-out cross-validation across eight participants.The CNN results were compared with NN1, NN2, and NN3 to assess the deep-learning approach.Source: §6 · pp. 6–7
Multi-level stress recognition
- 56.52% mean accuracy for three-class recognition with the CNN exceeded NN1 (51.89%), NN2 (51.02%), and NN3 (53.65%).The CNN result had SD=17.58%; NN1 performed better on low-stress predictions, while the CNN did better on no-stress and high-stress predictions.Source: §6 · p. 7
Binary stress recognition
- 84.59% mean accuracy for binary stress recognition with the CNN exceeded NN1 (74.89%), NN2 (74.81%), and NN3 (77.31%).The CNN result had SD=19.34%.Source: §6 · p. 7
7. Discussions
The authors report higher CNN accuracy than the best shallow network for both binary and three-class recognition, although the differences were not statistically significant. They present breathing variability spectrograms and low-cost thermal imaging as components of a non-contact stress-recognition approach.
Results and interpretation
- 84.59% binary and 56.52% multi-class accuracy exceeded the best shallow-network results of 77.31% and 53.65%, respectively, but differences were not significant.The authors suggest the relatively small participant count may relate to the lack of significance.Source: §7 · p. 7
Contributions
- The proposed respiration variability spectrograms capture breathing changes from sudden short-lasting shifts to longer-lasting respiration changes.The authors describe the dataset as labelled with stress levels from structured multi-level stress-induction tasks.Source: §7 · p. 7
Implications
- The authors report above-chance three-class recognition from respiration signals alone and suggest combining breathing variability with other signals may improve recognition.They attribute some of the high performance to breathing variability rather than breathing rate alone.Source: §7 · p. 7
Limitations
- The authors identify self-report ground truth, rating normalisation, and inter-person variability as areas where further improvement could be obtained.They mention continuous ratings over each session as one possible refinement.Source: §7 · p. 7
8. Conclusion
DeepBreath combines a new breathing-dynamics signature, data-augmented deep feature learning, and low-cost thermal imaging for stress recognition. Its multi-level, multi-task evaluation reports high performance under unconstrained conditions.
8. Conclusion
- The multi-level, multi-task stress-induction study reports high automatic stress-detection performance under unconstrained conditions, including verbal expressions and movement.The conclusion describes evaluation across a multi-level, multi-task study but gives no numeric performance values here.Source: §8 · p. 7
- DeepBreath combines a new bi-dimensional signature of breathing dynamics with data-augmented deep feature learning and low-cost thermal imaging.Source: §8 · p. 7
- The authors identify this as the first deep-learning approach applied to breathing-dynamic patterns.Source: §8 · p. 7