PaperlayerView exact revision on arXiv

Source-linked AI summary

Edge AI on Constrained Devices for Binary Sleep-Wake Classification in Dynamic Environments

Stefan Reitmann, Lena Oden

arXiv:2609.29163v1cs.LG

TL;DR

Mobile motion artifacts and embedded-device resource limits complicate reliable sleep-wake detection. The paper implements a multimodal, on-device Edge AI pipeline that combines inertial and visual analysis. It reports 96.5% test accuracy for motion analysis and 89% for pose classification, while noting limits tied to pose-dependent visual detection.

  • Problem

    Motion artifacts in mobile environments and limited compute and energy complicate reliable sleep-wake detection on wearable and IoT devices.

    Source: §I · p. 1

  • Method

    An ESP32-S3 system processes inertial head-motion data and visual pose input using a sequential strategy that activates vision after sustained motionlessness.

    Source: §I · p. 1, §V · p. 6, D. Sensor Fusion & Decision Logic · p. 7

  • Results

    96.5% test accuracy across 1,092 evaluation windows for the motion model, and 89% test accuracy on 227 images for the visual model.

    Source: 2) Performance · p. 6

  • Takeaways & Limitations

    The ESP32-S3 system demonstrates binary sleep-wake detection on resource-constrained embedded hardware in dynamic environments.

    Source: §VI · p. 7

    Visual detection depends on pronounced head poses during sleep, limiting applicability when those poses do not occur; subtle eye closure is less reliably captured.

    Source: E. Deployment & System Validation · p. 7

Abstract

from arXiv · show

This paper presents an Edge AI-based system for detecting sleep and wake states in non-stationary mobile environments using resource-constrained embedded hardware. Conventional approaches relying on accelerometer-based activity metrics are highly susceptible to motion and vibration artifacts and are limited by strict compute and energy budgets of wearable and IoT devices. To address these challenges, a multimodal pipeline is designed and implemented on an ESP32-S3 microcontroller. The system combines inertial sensing for head movement analysis and visual pose classification. A dual-core architecture with FreeRTOS enables parallel execution of real-time data acquisition and on-device inference. Sleep detection follows a two-stage strategy: low-movement detection over a temporal window, followed by visual validation of poses. Experimental results show accuracies of 96.5% for motion-based detection and 89% for pose classification, yielding robust binary sleep-wake classification. Field tests confirmed feasibility in representative mobile scenarios. The results demonstrate that privacy-preserving, local sleep detection is achievable on edge hardware through careful co-design, while highlighting limitations in sensing intrusiveness, dataset scale, and system integration.

I. INTRODUCTION

Sleep-wake detection in mobile settings must contend with motion artifacts as well as the limited compute and energy budgets of embedded devices. This work develops an on-device Edge AI system and frames its evaluation around robustness, efficiency, and sensor fusion.

I. INTRODUCTION

  • Motion and vibration in mobile settings can overlap with relevant physiological signals, complicating reliable sleep-wake discrimination.The challenge is especially acute in vehicles and bicycle trailers.Source: §I · p. 1
  • Wearable and IoT devices have limited compute and energy, making complex models impractical while accuracy must be maintained.Source: §I · p. 1
  • The paper develops an embedded system that processes motion and visual signals locally, avoiding reliance on cloud computation.The stated goals include privacy-preserving and energy-efficient operation.Source: §I · p. 1
  • The evaluation asks how to mitigate motion artifacts, balance accuracy with computational efficiency, and assess whether sensor fusion improves robustness over motion-only approaches.It considers classification accuracy, computational cost, energy consumption, and robustness to motion artifacts.Source: §I · p. 1

1) Embedded Systems and Edge AI:

  • Embedded systems operate under processing, memory, and energy constraints; Edge AI processes data locally, while RTOS scheduling supports concurrent sensing and inference.The paper identifies FreeRTOS as an example of an RTOS used to manage such tasks.Source: 1) Embedded Systems and Edge AI · pp. 1–2

2) Sensor Systems and Multimodal Sensing:

The reviewed work connects mobile sensing challenges with multimodal inference, embedded model deployment, sleep and drowsiness detection, and signal processing for noisy environments.

2) Sensor Systems and Multimodal Sensing:

  • IMUs and MPUs track movement and orientation in real time, but external vibration and environmental motion disturb their signals.Combining inertial data with optical, heart-rate, or contextual sensing is presented as a way to support inference where EEG or ECG may be impractical,.Source: 2) Sensor Systems and Multimodal Sensing · p. 2

3) TinyML and On-Device Learning:

  • TinyML targets machine-learning deployment on resource-constrained microcontrollers, using techniques such as quantization, pruning, and simpler architectures,, [9].TensorFlow Lite for Microcontrollers is cited as a deployment framework [10], alongside work on embedded deep-learning toolchains,.Source: 3) TinyML and On-Device Learning · p. 2

4) Sleep and Drowsiness Detection:

  • Traditional activity-based sleep and drowsiness metrics can be unreliable under external motion, while newer approaches combine wearable, smartphone, and contextual data –.Automotive work also uses facial landmarks and eye-state classification,, and studies report predictive value for heart-rate signals in fatigue assessment, [19].Source: 4) Sleep and Drowsiness Detection · p. 2

5) Signal Processing in Dynamic Environments:

  • Dynamic environments introduce noise and artifacts, and signal-processing methods including adaptive filtering, sliding windows, and frequency-domain transforms are used to separate signals from disturbances.Prior work reports that adaptive filtering can improve sleep detection under noisy conditions,, and preprocessing affects downstream feature extraction and model performance.Source: 5) Signal Processing in Dynamic… · p. 2

II. REQUIREMENTS ANALYSIS AND SYSTEM DESIGN

The design targets mobile, stimulation-rich settings where changing conditions make biosignals non-stationary. Its evaluation uses a car and a cargo-bike setup, linked to child sleep monitoring and adult driver fatigue detection.

1) Environments with Unsteady Conditions:

  • An unsteady environment is a mobile or stimulation-rich setting where motion, vibration, or changing light and noise make biosignals non-stationary.Such disturbances can produce artifacts and limit approaches designed for stationary settings.Source: 1) Environments with Unsteady Conditions · p. 2

2) Experimental Environments and Application Scenarios:

  • The evaluation considers a car with low-to-medium-frequency vibration and intermittent disturbances, and a cargo bike with strong vibration, high-frequency shocks, and substantial subject-sensor motion.The cargo-bike setup serves as a robustness stress test; Fig. 1 depicts the car and cargo-bike measurement setups.Source: Figure 1 · p. 2, 2) Experimental Environments and… · p. 3
  • The motivating use cases are monitoring children’s sleep during transit and detecting reduced alertness or microsleep in adult drivers.The stated applications include routine planning and longitudinal sleep analysis for children, and early warning and intervention strategies for drivers.Source: 2) Experimental Environments and… · p. 3

B. Criteria for Sleep State Detection

The section frames sleep detection around recognized sleep states and their characteristic features, while emphasizing that short transit episodes and age-related variation affect which phases may be observed.

1) Sleep Phases and Feature Extraction:

  • Sleep-state criteria distinguish wakefulness, non-REM sleep including light and deep stages, and REM sleep using characteristic physiological features.The text notes that EEG, ECG, and other measurement methods document these features.Source: 1) Sleep Phases and Feature Extraction · p. 3

2) Duration of Sleep Phases:

  • The work focuses on short sleep episodes or power naps, typically 10–30 minutes and primarily involving N1 and N2, while N3 may occur in longer 30–90 minute durations.REM sleep is usually reached only in extended sleep periods.Source: 2) Duration of Sleep Phases · p. 3
  • Sleep architecture varies with age; children have shorter cycles and higher REM proportions, while adults typically have 90–120-minute cycles with reduced REM and deep-sleep shares.The section identifies N1–N3 as most relevant to the applications while noting that REM-related characteristics may still occur in short episodes.Source: 2) Duration of Sleep Phases · p. 3

III. TECHNICAL IMPLEMENTATION AND SYSTEM ARCHITECTURE

The ESP32-S3 platform combines inertial and camera sensing with local processing, while FreeRTOS separates sensor acquisition from analysis across its cores.

A. Hardware Platform

  • An ESP32-S3 battery-powered platform performs sensing, preprocessing, and inference locally; its two-cell supply can support several hours of runtime depending on workload and duty cycle.Continuous wireless communication or camera use significantly increases current consumption.Source: §III · p. 3, A. Hardware Platform · p. 3

B. Sensor Configuration

  • The prototype pairs a 100 Hz MPU6050 for motion sensing with a 0.2 Hz OV3660 camera; optical pulse and environmental sensors were unsuitable for reliable inference in unsteady conditions.The pulse sensor was unstable under motion, while microphone, temperature, and light signals were affected by ambient noise and changing external conditions.Source: B. Sensor Configuration · pp. 3–4

C. Task Parallelization

  • Modular sensor and processing classes, queue-based communication, and mutex-protected access support maintainability and robust real-time operation.A central task manager initializes tasks, assigns cores, and controls execution.Source: C. Task Parallelization · p. 4
  • FreeRTOS assigns periodic sensing to one core and data management, storage, and decision logic to the other, separating time-critical acquisition from heavier processing.The camera task can be suspended when image capture is unnecessary, reducing resource use.Source: C. Task Parallelization · p. 4, Figure 3 · p. 4

D. Data Buffering and Synchronization

The system buffers and synchronizes sensor data for either storage or inference, with head-mounted inertial sensing and scenario-specific camera positioning.

D. Data Buffering and Synchronization

A. Physical Setup and Sensor Positioning

B. Data Aggregation and Preparation

Data preparation targeted binary head-motion and rest classification across automotive and cargo-bike scenarios, with inertial signals segmented into one-second windows.

B. Data Aggregation and Preparation

1) Inertial Dataset:

  • 124,343 samples at 100 Hz were collected from two adults and two children, including 76,686 no-intentional-motion and 47,657 motion samples.Source: 1) Inertial Dataset · p. 5
  • One-second sliding windows yielded 5,458 segments—3,406 motion and 2,052 rest—after edge trimming.Automotive data contributed 68,586 no-motion and 37,657 motion samples; cargo-bike data contributed 8,100 and 10,000, respectively.Source: 1) Inertial Dataset · p. 5

2) Camera Dataset:

The camera dataset uses facial-feature presence in a fixed ROI as an indirect pose cue, within a pipeline that gates vision on sustained motionlessness and deploys lightweight models on the ESP32-S3.

2) Camera Dataset:

  • The 1,178-image car dataset covered awake and sleep states across front and rear seats; a 96 × 96 ROI uses facial-feature presence as an indirect pose cue.Forward or lateral sleep-related head tilts can make facial features disappear, so the proxy is best suited to setups with limited movement and stable ROI positioning.Source: 2) Camera Dataset · p. 5, Figure 7 · p. 5

V. EDGE AI IMPLEMENTATION

  • The sequential decision framework activates resource-intensive vision only after sustained motionlessness, balancing computational efficiency with detection accuracy in mobile scenarios.Source: §V · p. 6

A. Training Pipeline and Optimization

B. Motion Detection Model (MPU6050)

  • An MLP classifies one-second motion windows from 700-dimensional vectors containing triaxial accelerometer and gyroscope features plus rotational energy; no explicit filtering was applied due to computational costs.Initial calibration subtracts rest-state offsets, and the rotational energy norm is an additional feature.Source: B. Motion Detection Model (MPU6050) · p. 6

1) Architecture:

The section describes compact inertial and visual classifiers and reports strong motion-classification performance. Both models use binary outputs, with architectures suited to on-device inference.

1) Architecture:

2) Performance:

  • 96.5% test accuracy was achieved by the MPU model across 1,092 evaluation windows.The confusion matrix is presented in Table IV.Source: 2) Performance · p. 6, Table IV · p. 6
  • The MPU model achieved F1-scores of 0.97 for no motion and 0.95 for motion, with an overall ROC-AUC of 0.99.Source: 2) Performance · p. 6

1) Architecture:

  • The camera model processes augmented 96×96 grayscale face regions with three convolutional blocks, global average pooling, and a sigmoid output.The blocks use 8, 16, and 24 filters, respectively.Source: 1) Architecture · p. 6

2) Performance:

The camera classifier achieved 89% test accuracy, while sequential sensor fusion activates vision after inactivity. Deployment tests found reliable wake detection, with visual sleep classification still limited by pose cues.

2) Performance:

  • 89% test accuracy was achieved by the camera model on 227 images.Table V presents its confusion matrix; macro-averaged F1 was 0.87, with sleep and awake F1-scores of 0.83 and 0.91.Source: 2) Performance · pp. 6–7, Table V · p. 7

D. Sensor Fusion & Decision Logic

  • The fusion logic continuously monitors inertial motion and activates the camera CNN only after sustained inactivity, then uses its prediction and a confidence threshold to validate sleep.An optional additional verification stage is described.Source: D. Sensor Fusion & Decision Logic · p. 7, Figure 10 · p. 7

E. Deployment & System Validation

  • The int8 MPU model occupies approximately 17 KB and averages 2.39 ms per inference; the int8 camera model occupies 11 KB and averages 2.7 s.Their TensorArena allocations are 16 KB and 512 KB, respectively.Source: E. Deployment & System Validation · p. 7
  • Wake states with head movement were detected in all 10 scenarios, and camera activation consistently followed detected inactivity.Functional tests used 30-second scenarios with two participants and DECISION_FRAME=3 and DECISION_SAMPLE=5000.Source: E. Deployment & System Validation · p. 7
  • Visual detection reliably captured pronounced forward or lateral head tilts, but subtle eye closure without pose change was less reliably captured.The reported pose detection was 9/10 in the car and 8/10 on the bike.Source: E. Deployment & System Validation · p. 7
  • The fusion logic operated as intended and enabled reliable wake detection, while sleep classification—particularly from visual cues—needs improvement and requires larger, more diverse evaluation data.Source: E. Deployment & System Validation · p. 7

VI. CONCLUSION & OUTLOOK

The conclusion presents ESP32-S3 Edge AI as feasible for binary sleep-wake detection in dynamic environments, while identifying pose dependence and dataset breadth as remaining constraints.

VI. CONCLUSION & OUTLOOK

  • The ESP32-S3 system combines heterogeneous sensors and lightweight models for binary sleep-wake inference in dynamic environments.The paper describes its operation as energy-efficient and suitable for long-term deployment.Source: §VI · p. 7
  • Inertial head-motion analysis distinguishes rest and activity under non-stationary conditions, while visual pose analysis remains dependent on sleep-related pose changes.The lightweight CNN is described as feasible on an MCU.Source: §VI · p. 7
  • Future work calls for larger, more diverse datasets, improved usability through wireless architectures and interfaces, and stronger visual representations for finer-grained sleep and fatigue detection.Source: §VI · p. 7
Loading 2609.29163v1…