Source-linked AI summary
Action2Activity: Recognizing Complex Activities from Sensor Data
Ye Liu, Liqiang Nie, Lei Han, Luming Zhang, David S Rosenblum
TL;DR
The paper addresses the limited bridging of simple actions and complex activities, whose recognition requires modeling temporal structure, activity relatedness, and discriminative features. It combines temporal pattern mining with adaptive multi-task learning, and experiments on real-world data report significant gains over state-of-the-art methods. The authors identify error propagation from low-level action recognition as future work.
Problem
Complex activities involve rich temporal action relations, relatedness across activities, and many nondiscriminative features, while prior work has done relatively little to bridge actions and activities.
Method
The approach mines frequent temporal patterns for activity representation and uses adaptive multi-task learning to model activity relatedness and select task-specific and task-sharing features.
Results
Extensive experiments on real-world data show significant gains from the two components and their overall performance compared with state-of-the-art methods.
Takeaways & Limitations
Temporal patterns and adaptive multi-task learning together provide an effective scheme for recognizing activities from sensor data.
Takeaways & Limitations
The work leaves error propagation from low-level action recognition outputs for future investigation.
Abstract
from arXiv · showhide
As compared to simple actions, activities are much more complex, but semantically consistent with a human's real life. Techniques for action recognition from sensor generated data are mature. However, there has been relatively little work on bridging the gap between actions and activities. To this end, this paper presents a novel approach for complex activity recognition comprising of two components. The first component is temporal pattern mining, which provides a mid-level feature representation for activities, encodes temporal relatedness among actions, and captures the intrinsic properties of activities. The second component is adaptive Multi-Task Learning, which captures relatedness among activities and selects discriminant features. Extensive experiments on a real-world dataset demonstrate the effectiveness of our work.
1 Introduction
Complex activity recognition must represent temporal structure among actions, relatedness across activities, and discriminative features. The paper addresses these challenges with temporal pattern mining and adaptive multi-task learning.
- Real-world activities combine temporally related actions, including sequential, interleaved, and concurrent relations.Cooking illustrates actions that may occur sequentially or simultaneously.
- Activity recognition is challenging because action relations create semantic meaning, activities share patterns, and feature spaces contain nondiscriminative dimensions.
- The proposed approach mines frequent temporal patterns to characterize activities from action sequences.Patterns encode temporal relations among actions and provide a mid-level activity representation.
- Adaptive multi-task learning models relatedness among activity-recognition tasks while identifying task-specific and task-sharing discriminant features.It also enables sharing training instances among tasks to alleviate insufficient training samples.
2 Related Work
Prior work largely recognizes isolated simple actions or models only simple relations among actions. The paper motivates richer representations because real activities contain overlapping actions and task-specific feature relevance.
- Earlier systems recognized isolated daily actions using simple features and classifiers.One cited example classified eight actions, while another recognized six actions for music recommendation.
- Dynamical models such as HMMs and CRFs capture sequential relations but cannot characterize complex relations involving overlapping actions.Treating activities as purely sequential can lose information.
- Multi-task learning jointly learns related tasks and can improve generalization, particularly when training samples are insufficient.
- Activity recognition also requires selecting features that are discriminative for specific tasks rather than forcing every task to use identical features.Group Lasso-based approaches combine shared and task-specific feature selection.
3 Temporal Pattern Mining
Temporal pattern mining represents activities through frequent sets of actions linked by temporal relations. The method constructs a joint pattern feature space and prunes infrequent patterns during discovery.
- Each activity is represented as a start-time-ordered sequence of actions, with every action carrying an identifier, start time, and end time.
- A temporal pattern is a set of actions plus a relation matrix encoding pairwise temporal relatedness through Allen’s interval logic.
- The subpattern partial order enables pruning because every subpattern of a frequent temporal pattern must also be frequent.The stated property ensures that the mining algorithm does not miss frequent patterns.
- Pattern support is normalized observation time within a sliding window and is interpreted as the pattern’s observation probability for an activity.
- A pattern is frequent when its support meets the minimum support threshold.
- The algorithm iteratively generates higher-dimensional patterns, removes infrequent patterns, and stops when no further frequent patterns are found.The resulting patterns form a joint feature space whose entries are activity-conditioned support values.
4 Adaptive Multi-Task Learning
The approach formulates each activity-recognition problem as a task and jointly learns task relatedness while selecting discriminative temporal-pattern features. Adaptive optimization learns task relations through a positive semi-definite matrix and uses group sparsity to balance shared and task-specific features.
- 4 Adaptive Multi-Task Learning: Activities share patterns, but their high-dimensional temporal-pattern features are not all discriminative, motivating joint relatedness modeling and feature selection.Similar activities can share temporal patterns, while only a subset of patterns is predictive for recognition.
- 4 Adaptive Multi-Task Learning: The training set contains M activity-recognition tasks, with each sample represented by a temporal-pattern feature vector and an M-dimensional label vector.Each sample corresponds to one activity label among the M tasks.
- 4 Adaptive Multi-Task Learning: The objective combines empirical error, adaptive task relatedness, generalization control, and a group Lasso penalty for automatic feature selection.The task-relation matrix Ω is positive semi-definite and learned rather than uniformly or manually specified.
- 4 Adaptive Multi-Task Learning: The ℓ2,1-norm groups each feature’s weights across tasks, encouraging shared feature selection while retaining activity-specific features.This penalty reflects the assumption that only a small set of features predicts each recognition task.
- 4 Adaptive Multi-Task Learning: The convex optimization is solved by alternative optimization, fixing Ω or W in turn; the W-subproblem uses FISTA.With Ω fixed, the rewritten problem is unconstrained and convex, while Ω is optimized with W fixed.
5 Experiments
Experiments on the Opportunity dataset evaluate temporal-pattern representations and learning models for complex activity recognition. Results support higher-order temporal patterns and adaptive multi-task learning, with efficiency and scalability analyses indicating practical feasibility.
- Dataset: The Opportunity dataset records five complex activities using 72 body and object sensors across four subjects.
- Temporal Patterns: Temporal-pattern representations outperform bag-of-actions, and higher-order patterns further improve activity recognition under the same aMTL model.Bag-of-actions with aMTL achieves 93.3% accuracy, while significance tests report p-values below 0.05 for representation comparisons and higher-pattern improvements.
- Learning Model Comparison: aMTL improves overall accuracy by 2%-4% over MTL and by 3%-6% over GL by jointly learning activity-sharing and activity-specific temporal features.
- Learning Model Comparison: All pairwise p-values between aMTL and the baselines are smaller than 0.05 across temporal-pattern dimensions.
- Pattern Dimension: Accuracy tends to stabilize when pattern dimension k exceeds 3, while larger patterns increase feature dimensionality and extraction cost.
- Efficiency and Scalability: The alternating optimization usually converges within 20 iterations, with per-iteration cost O((N + M)DM) for FISTA-based optimization and O(DM^2 + M^3) for the closed-form step.
6 Conclusion and Future Work
The paper presents a two-component sensor-based activity-recognition scheme combining temporal pattern mining with adaptive multi-task learning. Future work will address error propagation from low-level action recognition systems.
- Temporal pattern mining extracts frequent patterns from low-level actions, while adaptive multi-task learning captures relatedness among activities.
- Extensive experiments on real-world data show gains for both components and for their combined performance against state-of-the-art methods.
- Future work will address error propagation in the outputs of low-level action recognition systems.