Source-linked AI summary
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, Tarek Abdelzaher
TL;DR
Mobile sensing must estimate or classify from noisy time-series data, while physical models and hand-crafted features are difficult to customize robustly. DeepSense combines convolutional and recurrent networks to learn modality interactions and temporal dynamics in a unified framework. Across three sensing tasks, it outperforms state-of-the-art methods and remains feasible for smartphone deployment.
Problem
Noisy sensor measurements complicate physical-model-based estimation, while hand-crafted features are time-consuming to design for diverse noise patterns and user behaviors.
Method
DeepSense integrates CNNs and RNNs to learn local, cross-modality, and temporal relationships from sensor measurements for regression and classification tasks.
Results
DeepSense outperforms state-of-the-art methods across car tracking, heterogeneous activity recognition, and biometric user identification, with moderate energy consumption and low latency on mobile devices.
Takeaways & Limitations
The framework provides one customizable approach for regression-oriented and classification-oriented mobile sensing while remaining implementable on mobile devices.
Takeaways & Limitations
Drastic changes in the physical environment may require retraining DeepSense with new data.
Abstract
from arXiv · showhide
Mobile sensing applications usually require time-series inputs from sensors. Some applications, such as tracking, can use sensed acceleration and rate of rotation to calculate displacement based on physical system models. Other applications, such as activity recognition, extract manually designed features from sensor inputs for classification. Such applications face two challenges. On one hand, on-device sensor measurements are noisy. For many mobile applications, it is hard to find a distribution that exactly describes the noise in practice. Unfortunately, calculating target quantities based on physical system and noise models is only as accurate as the noise assumptions. Similarly, in classification applications, although manually designed features have proven to be effective, it is not always straightforward to find the most robust features to accommodate diverse sensor noise patterns and user behaviors. To this end, we propose DeepSense, a deep learning framework that directly addresses the aforementioned noise and feature customization challenges in a unified manner. DeepSense integrates convolutional and recurrent neural networks to exploit local interactions among similar mobile sensors, merge local interactions of different sensory modalities into global interactions, and extract temporal relationships to model signal dynamics. DeepSense thus provides a general signal estimation and classification framework that accommodates a wide range of applications. We demonstrate the effectiveness of DeepSense using three representative and challenging tasks: car tracking with motion sensors, heterogeneous human activity recognition, and user identification with biometric motion analysis. DeepSense significantly outperforms the state-of-the-art methods for all three tasks. In addition, DeepSense is feasible to implement on smartphones due to its moderate energy consumption and low latency
1. INTRODUCTION
DeepSense addresses noisy time-series mobile sensing across regression and classification tasks with one customizable framework. It combines CNNs and RNNs to learn sensor interactions, temporal relationships, and task-relevant representations, outperforming prior methods while remaining feasible on mobile devices.
- Motivation: Mobile sensing applications estimate continuous or categorical outputs from noisy time-series measurements.The paper frames these as regression and classification problems.
- Challenges: Regression methods rely on physical models, but nonlinear and temporally correlated sensor noise makes accurate modeling difficult.Acceleration errors can accumulate through double integration when estimating displacement.
- Challenges: Classification pipelines use hand-crafted features whose design is time-consuming and may not generalize across sensor noise patterns and heterogeneous users.The human activity recognition task specifically highlights poor generalization to unseen users.
- Approach: DeepSense integrates CNNs and RNNs to learn local modality interactions, cross-modality global interactions, and temporal relationships from sensor intervals.Frequency representations are processed by CNNs before interval representations are modeled sequentially by an RNN.
- Evaluation: Across car tracking, heterogeneous activity recognition, and biometric user identification, DeepSense outperforms state-of-the-art methods and simpler variants.The framework reports smaller tracking error, large-margin classification improvements, and superiority over its simplified variants.
- Deployment: DeepSense remains implementable on smartphones, with moderate energy consumption and low overhead or latency across the evaluated tasks.Feasibility was demonstrated on two types of smart devices.
2. RELATED WORK
Prior deep-learning approaches address selected modalities or task types but often omit either temporal modeling or multimodal integration. DeepSense is presented as a unified framework spanning regression-based and classification-based mobile sensing problems.
- Existing approaches: Deep-learning architectures have been applied broadly, but their relationship modeling capabilities differ across input types.Residual networks focus on visual inputs, while LRCNs model video temporal structure.
- Existing approaches: LRCNs model temporal relationships but do not consider multimodal inputs, while multimodal DBMs merge modalities without modeling temporal relationships.Both omissions matter for mobile sensing tasks requiring multiple sensors and time-series inputs.
- Mobile sensing literature: Existing mobile-sensing studies include noisy audio sensing, heterogeneous activity recognition, and biometric gait analysis using different deep-learning models.DeepEar, RBM, MultiRBM, and IDNet target particular sensing applications.
- Positioning: DeepSense is described as the first framework to directly solve both regression-based and classification-based mobile sensing problems in one unified manner.The cited related methods focus on classification-oriented tasks only.
3. DEEPSENSE FRAMEWORK
DeepSense converts multivariate sensor streams into interval-wise frequency representations, processes each modality and their fusion with CNNs, models intervals with stacked GRUs, and produces task-specific outputs.
- Input representation: Each sensor provides a measurement matrix and timestamps, which DeepSense splits into non-overlapping intervals of width τ.The resulting interval sequence contains T intervals.
- Input representation: Fourier transforms convert each interval into frequency-domain magnitude and phase features that form the CNN input tensors.For each sensor, the tensor has dimensions d(k) × 2f × T.
- Convolutional layers: Individual convolutional subnets learn local interactions within each sensor modality before a merge subnet learns interactions across all K sensors.The convolutional layers are organized hierarchically from modality-specific processing to cross-sensor fusion.
- Convolutional layers: The merged convolutional representations are flattened, combined with interval width, and passed through recurrent layers to model relationships over time.The CNN outputs and τ become the recurrent-layer inputs.
- Recurrent layers: DeepSense uses stacked GRUs because they increase capacity while supporting incremental stream processing more efficiently than bidirectional GRUs.The paper uses two GRU layers and applies dropout and recurrent batch normalization.
- Output layer: For regression, a shared fully connected output layer decodes recurrent vectors into physical quantities at each interval.The output layer learns Wout and bout to transform each recurrent representation.
- Output layer: For classification, recurrent features are averaged across intervals and passed to a softmax layer for category probabilities.Averaging is used instead of a more sophisticated attention-based aggregation.
4. TASK-SPECIFIC CUSTOMIZATION
DeepSense is customized by configuring sensor inputs, task type, preprocessing, and cost function. The framework is instantiated across car tracking, heterogeneous activity recognition, and user identification with task-specific objectives.
- General Customization Process: DeepSense customization begins by identifying sensor inputs, preprocessing them into tensors, and selecting regression or classification output layers.The default configuration requires setting the number of inputs, preprocessing measurements, and identifying the task type.
- General Customization Process: The cost function may be customized with penalty or regularization terms, while default objectives use mean square error for regression and cross-entropy for classification.The model cost combines a loss function with penalty terms whose importance is controlled by λj.
- General Customization Process: Sensor measurements are aligned, chunked, Fourier-transformed, and stacked into d(k) × 2f × T tensors for each sensor.Here, d(k) is the measurement dimension, f the frequency-domain dimension, and T the number of time intervals.
- Car tracking with motion sensors (CarTrack): CarTrack uses accelerometer, gyroscope, and magnetometer inputs for regression, with GPS-derived displacement converted into a 2D Gaussian distribution for training.A Kalman filter is applied because GPS displacement labels are noisy.
- Car tracking with motion sensors (CarTrack): CarTrack uses negative log likelihood with an additional cosine-similarity penalty when predicted displacement direction exceeds a predefined angular margin.The penalty reflects the experiment’s emphasis on predicting the correct direction.
- Heterogeneous Human activity recognition (HHAR) and User Identification (UserID): HHAR uses accelerometer and gyroscope measurements for leave-one-user-out classification with the default cross-entropy objective.UserID likewise classifies identities from accelerometer and gyroscope measurements using cross-entropy.
5. EVALUATION
DeepSense is evaluated on CarTrack, HHAR, and UserID using task-specific datasets and baselines, with accuracy, energy, and latency as evaluation dimensions. It achieves substantially better task performance than competing methods while maintaining moderate energy consumption and low latency on mobile devices.
- Experimental setup: The evaluation covers CarTrack tracking, heterogeneous human activity recognition, and biometric UserID classification using collected data or existing datasets.DeepSense is compared with state-of-the-art task-specific algorithms and three simplified DeepSense variants.
- CarTrack: DeepSense reduces CarTrack tracking error by an order of magnitude compared with the sensor-fusion algorithm.The paper attributes this improvement mainly to learning the composition of the noise model and physical laws.
- CarTrack: 40.43 ± 5.24m mean absolute error is achieved on CarTrack, enabling map-aided trajectory tracking with a high probability of accuracy.The reported error is described as almost half of a traditional 80m × 80m city block.
- CarTrack: Longer driving distances generally increase mean absolute error, but the increase is not linear; map-aided accuracy can improve slightly with distance.The paper links this pattern to early direction uncertainty and longer trajectories containing segments useful for map matching.
- HHAR: DeepSense-based HHAR models outperform other baseline algorithms by at least 10%, reaching accuracy 0.942 ± 0.032 and macro F1 0.931 ± 0.041.The reported gains are associated with automatically learned features that capture local, cross-sensor, and temporal structure.
- Mobile feasibility: All DeepSense-based models show moderate energy consumption and low latency across the evaluated mobile tasks and devices.For CarTrack, energy is slightly below 1-Hz GPS sampling on Nexus 5, with runtimes measured in microseconds; HHAR predictions also have moderate energy and low latency.
6. DISCUSSION
DeepSense is designed as a unified framework for diverse mobile sensing tasks, with customization options that trade accuracy, energy, and task specificity. Its evaluation scope and retraining needs define practical boundaries for deployment.
- Framework customization: DeepSense uses one framework across different mobile sensing tasks without tailoring its architecture to each task.Task-specific architectural changes can provide additional performance gains.
- Framework customization: Separating the learned noise model from known physical laws can turn DeepSense into a denoising component for regression tasks.The original framework learns the composition of noise models and physical laws directly from data.
- Framework customization: Removing design components can reduce energy consumption with acceptable accuracy degradation for particular tasks.Individual convolutional subnets model within-sensor relationships, while merge convolutions and stacked RNNs provide cross-sensor and temporal capacity.
- Scope and limitations: The evaluation focuses mainly on motion sensors, although DeepSense can be applied directly to microphones, Wi-Fi signals, barometers, and light sensors.The paper identifies applying the framework to new smart-device applications as requiring further study.
- Scope and limitations: Drastic changes in the physical environment may require retraining DeepSense with new data.An existing trained framework can initialize this process and aid optimization and generalization.
7. CONCLUSION
The paper introduces DeepSense as a unified framework for mobile sensing and computing. Across three representative tasks, it reports strong performance against state-of-the-art baselines while maintaining mobile feasibility through moderate energy consumption and low latency.
- 7. CONCLUSION: DeepSense integrates convolutional and recurrent neural networks to learn sensor relationships for regression and classification tasks.It models physical laws and noise for regression while extracting local, global, and temporal features for classification.
- 7. CONCLUSION: DeepSense outperformed state-of-the-art baselines by significant margins across three representative mobile sensing tasks.The tasks cover car tracking, heterogeneous human activity recognition, and user identification with biometric motion analysis.
- 7. CONCLUSION: DeepSense remained feasible for mobile deployment through moderate energy consumption and low latency on mobile and embedded platforms.The framework variants also provide guidance for adaptation and customization across applications.