Source-linked AI summary
Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
Md. Milon Islam, Sheikh Nooruddin, Fakhri Karray, Ghulam Muhammad
TL;DR
HAR research needs a consolidated view of how CNN architectures support recognition across wearable and stationary sensing modalities. This review surveys CNN-based systems, datasets, hyperparameters, strengths, weaknesses, and challenges across four device categories, reporting modality-dependent performance and open issues. It concludes that composite activities and contextual information remain important boundaries for current systems.
Problem
HAR research spans many CNN-based approaches and sensing modalities, but their performances, strengths, weaknesses, hyperparameters, datasets, and challenges require consolidated review.
Method
The paper reviews recent CNN-based HAR systems across multimodal sensors, smartphones, radar, and vision, analyzing architectures, hyperparameters, benchmark datasets, and open issues.
Results
The review reports that the highest accuracy exceeded 99% for reviewed multi-sensing-device and radar systems, while comparisons remain difficult because datasets differ.
Takeaways & Limitations
CNN-based HAR is organized into four sensing modalities with distinct deployment characteristics, and the review identifies challenges and future research directions for each.
Takeaways & Limitations
Current HAR systems lack sufficient contextual information and datasets for concurrent composite activities, limiting recognition of behavior whose meaning depends on setting or repetition.
Abstract
from arXiv · showhide
Human Activity Recognition (HAR) plays a significant role in the everyday life of people because of its ability to learn extensive high-level information about human activity from wearable or stationary devices. A substantial amount of research has been conducted on HAR and numerous approaches based on deep learning and machine learning have been exploited by the research community to classify human activities. The main goal of this review is to summarize recent works based on a wide range of deep neural networks architecture, namely convolutional neural networks (CNNs) for human activity recognition. The reviewed systems are clustered into four categories depending on the use of input devices like multimodal sensing devices, smartphones, radar, and vision devices. This review describes the performances, strengths, weaknesses, and the used hyperparameters of CNN architectures for each reviewed system with an overview of available public data sources. In addition, a discussion with the current challenges to CNN-based HAR systems is presented. Finally, this review is concluded with some potential future directions that would be of great assistance for the researchers who would like to contribute to this field.
1 Introduction
Human activity recognition identifies physical, body-part, object-related, and abnormal activities using sensing systems. This review focuses on CNN-based HAR, organizing recent systems by modality and discussing their performance, limitations, datasets, and future directions.
- Human activity recognition covers tasks such as walking, sitting, hand gestures, cooking, and detecting abnormal events.
- CNN-based HAR systems use input devices, data collection, communication, and computing or storage resources as successive architectural stages.
- The review specifically surveys CNN-based systems using multimodal sensors, smartphones, radar, and vision data.
- The survey analyzes system performance, strengths, weaknesses, CNN hyperparameters, benchmark datasets, challenges, and future research directions.
- The article describes its organization across paper selection, CNN-based systems, public datasets, open issues, future work, applications, and conclusions.
2 Selection of papers
The review selected recent CNN-based HAR research through manual searches of five scholarly sources using modality-specific keywords.
- The review considered works published during the last nine years and searched Google Scholar, PubMed, NCBI, CINAHL, and Web of Science.
- Searches used keywords covering CNN-based HAR, radar, vision, multimodal, smartphone systems, and public datasets.
3 Literature Review on Human Activity Recognition
The literature review groups CNN-based HAR systems by sensing modality and examines their architectures, challenges, and reported performance. Across modalities, practical issues include sensor correlation, device position, computation, contextual variation, and complex activities.
- Modalities: The review covers multimodal sensing devices, smartphones, radar signals, and image or video signals as four HAR categories.
- Multimodal sensing devices: Multimodal HAR must learn inter-modality correlations while preserving intra-modality information across sensors.
- Vision systems: 98.61% F1 Score was obtained by a system combining local spatial, global spatial, and temporal features in a real-time Raspberry Pi prototype.
- Smartphone sensor data: Smartphone systems address changing device positions using position-independent designs and data collected from bags, hands, and pockets.
- Smartphone sensor data: Smartphone deployment is constrained by GPU cost and power demands, motivating parallel hand-crafted and deep features or compact CNNs.
- Radar signal: Radar representations retain temporal correlations that may be poorly handled when two-dimensional radar matrices are treated as optical images.
- Image and video data: Video systems can reduce computation by classifying selected frames, but assigning one action becomes problematic when sequences contain multiple actions or groups.
4 Data Availability
The review surveys public HAR datasets spanning sensor-based and vision-based data, with variation in subjects, settings, spontaneity, activities, and sensing modalities.
- Dataset characteristics: The datasets include raw, transformed, or fused sensor data and generally involve more than 10 test subjects across different collection scenarios.The review selected datasets based on their frequency of use in HAR research.
- Dataset metadata: Dataset tables organize general characteristics alongside activity types, sample information, sensing points, modalities, and sensor positions.Smartphone-based datasets generally place the smartphone near the front thigh.
- Collection settings: Dataset entries distinguish laboratory and out-of-lab settings and whether activities are spontaneous or deliberately performed.Examples include WISDM v1.1 and v2.0 as out-of-lab but not spontaneous, and ActiveMiles as out-of-lab and spontaneous.
- Activity coverage: Common daily activities include standing, jogging, running, jumping, cycling, crouching, stair climbing, lying down, and walking.PAMAP2 and MSR Daily Activity also include activities not generally present in standard ADL datasets.
- Vision datasets: Large-scale YouTube datasets provide spontaneous, out-of-lab vision data, including collections with 1 million and 8 million activity videos.Their camera positions vary widely because the videos come from diverse recordings.
5 Discussions, Open Issues, and Future Works
The reviewed CNN-based HAR systems report strong results across sensing modalities, but comparisons remain difficult because datasets and evaluation settings differ. Open issues include computational cost, limited real-world validation, complex activities, contextual variation, and incomplete reproducibility.
- Reviewed-system performance: Accuracy above 99% was reported for multimodal-sensing and radar systems, while cross-system comparison remains difficult because the datasets differ.UCF101 and HMDB51 were frequently used for vision systems, whereas mhealth, skoda, and OPPORTUNITY were frequent datasets for multimodal systems.
- Deployment constraints: Several systems report high performance or useful feature extraction, but practical deployment is constrained by latency, computational complexity, noise, and limited real-time or user testing.Reported constraints include approximately eight-minute testing time, high training or classification time, large parameter counts, sensor noise, and untested real-time use with actual users.
- Open challenges: CNN-based HAR struggles with composite activities because available datasets rarely represent concurrent routines, which are harder to model than basic activities.The review identifies composite-activity datasets as significantly lacking.
- Open challenges: Real-world recognition is limited by inter-activity variation, contextual dependence, environmental interference, sensor constraints, and differences between experimental and natural settings.Examples include occlusion, lighting, noise, portability and wearability constraints, and context-dependent interpretations such as resting versus falling.
6 Conclusion
Human activity recognition supports recording, monitoring, analysis, and assistance of daily life across sensing modalities, with applications in healthcare, surveillance, and entertainment.
- Human activity recognition systems record general human activities through different sensing modalities.
- These systems support monitoring, analysis, and assistance of daily life through capable computing systems.
- Human activity recognition has applications in healthcare, surveillance, and entertainment systems.