Source-linked AI summary
PaStaNet: Toward Human Activity Knowledge Engine
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, Cewu Lu
TL;DR
Image-based activity understanding commonly maps images directly to activity concepts, leaving a substantial semantic gap. The paper builds PaStaNet and uses Activity2Vec with PaSta-based reasoning to infer activities from body-part states, reporting improvements across supervised, one-shot, and transfer settings.
Problem
Direct image-to-activity mapping faces a performance bottleneck because of the semantic gap, long-tail data distributions, and complex visual patterns.
Method
The paper constructs PaStaNet and uses Activity2Vec to extract part-level representations followed by PaSta-R to reason about activities from instance- and part-level states.
Results
The method improves mAP by 6.4 on HICO, 13.9 on one-shot HICO, 3.2 on V-COCO, and 4.2 on images-based AVA.
Takeaways & Limitations
PaSta provides reusable and transferable action components with interpretable activity representations, supporting activity understanding especially under few-shot conditions.
Abstract
from arXiv · showhide
Existing image-based activity understanding methods mainly adopt direct mapping, i.e. from image to activity concepts, which may encounter performance bottleneck since the huge gap. In light of this, we propose a new path: infer human part states first and then reason out the activities based on part-level semantics. Human Body Part States (PaSta) are fine-grained action semantic tokens, e.g. <hand, hold, something>, which can compose the activities and help us step toward human activity knowledge engine. To fully utilize the power of PaSta, we build a large-scale knowledge base PaStaNet, which contains 7M+ PaSta annotations. And two corresponding models are proposed: first, we design a model named Activity2Vec to extract PaSta features, which aim to be general representations for various activities. Second, we use a PaSta-based Reasoning method to infer activities. Promoted by PaStaNet, our method achieves significant improvements, e.g. 6.4 and 13.9 mAP on full and one-shot sets of HICO in supervised learning, and 3.2 and 4.2 mAP on V-COCO and images-based AVA in transfer learning. Code and data are available at http://hake-mvig.cn/.
1. Introduction
Image-based activity understanding often maps pixels directly to activity concepts, but the paper argues that fine-grained body-part semantics offer a more effective path toward an activity knowledge engine. PaStaNet, Activity2Vec, and PaSta-based reasoning support this approach across supervised and transfer settings.
- Direct pixel-to-activity mapping faces performance bottlenecks on large-scale benchmarks because activities involve long-tail distributions and complex visual patterns.
- PaStaNet encodes fine-grained human body-part states as reusable semantic components of activities.The knowledge base contains 118K+ images, 285K+ persons, 250K+ interacted objects, 724K+ activities, and 7M+ human part states.
- Activity2Vec extracts general part-level representations, while PaSta-R infers activities by combining instance- and part-level semantic states.
- 13.9 mAP is gained on one-shot HICO sets, reflecting PaSta reusability across different activities.The paper also reports interpretability because predicted activities can be related to what body parts are doing.
- 6.4 mAP and 5.6 mAP improvements are reported on HICO and HICO-DET, respectively, in supervised and transfer learning settings.
2. Related Works
Prior activity-understanding work includes image- and video-based approaches, but activity categories and dataset definitions vary substantially. This diversity makes knowledge transfer across datasets ineffective and motivates alternative representations.
- Image- and video-based activity understanding has improved with deep learning and large-scale datasets.
- Human activities have hierarchical structures and diverse verbs, making explicit category organization difficult.
- Differences in activity definitions across datasets make knowledge transfer ineffective.
- Existing approaches include holistic body-level, body-part-based, and skeleton-based methods.
3. Constructing PaStaNet
PaStaNet represents activities through fine-grained states assigned to ten human body parts. Its annotations combine crowdsourced and existing activity images, support multiple simultaneous states, and scale through assisted labeling and quality checks.
- PaSta decomposes the human body into ten parts and assigns each part descriptive states such as holding, pushing, watching, or eating something.
- Each body part can have multiple PaSta labels because a person may perform several actions simultaneously.For example, the head may eat, talk to, and look at something during eating while talking.
- PaStaNet combines crowdsourced human-centric images with images from existing activity datasets to improve generality.
- 76 PaSta candidates are selected using Normalized Point-wise Mutual Information between activities and candidate part states.
- Seed annotations for 10K images enable automatic initial labeling, after which additional annotators revise the results.Each image is annotated twice and checked by automatic procedures and supervisors; outliers are discarded for robust agreement.
- 118K+ images, 285K+ persons, 250K+ interacted objects, 724K+ activities, and 7M+ PaSta comprise the final PaStaNet.
4. Activity Representation by PaStaNet
PaStaNet replaces direct image-to-activity mapping with part-state recognition, Activity2Vec representations, and PaSta-based reasoning over a hierarchical activity graph.
- PaStaNet Paradigm: Conventional methods directly map images to activity concepts, whereas PaStaNet first recognizes human part states and then infers activities from them.The paradigm uses body-part and object features before reasoning over activity semantics.
- Activity2Vec: Activity2Vec extracts ten part-level PaSta representations from automatically generated body-part boxes and combines visual and language features.PaSta tokens are encoded with visual representations and BERT-derived language features before pooling into part representations.
- PaSta Recognition: PaSta recognition uses part relevance to emphasize informative body parts before predicting multilabel states for each part.The method can assign multiple states to one part and uses multiple sigmoid outputs for multilabel classification.
- Activity2Vec: PaSta representations combine visual features with language priors, including frozen BERT representations of part, verb, and object tokens.This combination is designed to bridge PaSta and activity semantics.
- Activity2Vec: The resulting part-level activity representation can support downstream tasks such as activity detection and captioning.The representation is intended as a general feature extractor for activity-related tasks.
- PaSta-based Activity Reasoning: PaSta-R uses part representations and object or image features to infer activity scores within a hierarchical activity graph.For human-object interactions it uses object features; for body-only motions it uses whole-image features.
5. Experiments
Experiments evaluate PaSta-based activity recognition, instance detection, transfer learning, and ablations across HICO, HICO-DET, V-COCO, and image-based AVA. Results show stronger performance, particularly when PaSta knowledge is available for few-shot and transfer settings.
- Instance-based Activity Detection: 46.3, 66.8, 32.0, 68.6, 56.2, and 65.8 mAP are achieved for head, arms, hands, hip, thighs, and feet PaSta recognition, respectively.These results use the PaStaNet* mode on HICO-DET.
- Instance-based Activity Detection: 13.9 mAP improvement on few-shot HICO-DET sets demonstrates strong reusability and transferability of PaSta representations.The PaStaNet mode also achieves a relative 16% improvement, while the hierarchical paradigm reaches 65.6 mAP with ground-truth PaSta in GT-PaStaNet* mode.
- Transfer Learning with Activity2Vec: 3.25 mAP improvement on HICO-DET Default Full reaches 20.28 mAP in transfer learning without using the excluded HICO-DET images and labels for pre-training.PaStaNet is pre-trained on 71K remaining images before fine-tuning.
- Ablation Study: Removing Part Attention decreases HICO-DET performance by 0.21 mAP, while replacing PaSta BERT features with Gaussian noise, Word2Vec, or GloVe yields 20.80, 21.95, and 22.01 mAP.The TIN+PaSta*-Linear baseline is 22.12 mAP; converting the PaSta triplet into a sentence gives 22.26 mAP.
6. Conclusion
The paper presents PaStaNet as a body-part-level activity representation and combines Activity2Vec with PaSta-R to improve supervised and transfer activity understanding. It reports especially strong gains under few-shot conditions and identifies spatio-temporal PaSta as future work.
- Conclusion: PaStaNet provides body part-level activity representations intended to bridge the semantic gap between pixels and activities.The authors describe this as a step toward a human activity knowledge engine.
- Conclusion: Activity2Vec extracts part-level representations, while PaSta-R performs part-based activity reasoning.Together, these methods are used with PaStaNet for activity understanding.
- Conclusion: The method improves supervised and transfer learning performance, especially under few-shot circumstances.The conclusion states that PaStaNet brings interpretability and new possibilities for activity understanding.
- Conclusion: Future work will enrich PaStaNet with spatio-temporal PaSta.The paper does not report results for this planned extension.
Appendices
The appendices visualize PaSta feature representations and document PaStaNet’s dataset characteristics. The accompanying table identifies the dataset’s image sources.
- PaSta Representations: t-SNE visualizations show PaSta representations for foot and hand features.Human body part patches with different colored borders replace embedding points in the visualization.
- Annotator Backgrounds: Annotator backgrounds are presented as a separate appendix figure.The figure is labeled Figure 7.
A. Dataset Details
PaStaNet organizes human activity knowledge around body-part states, annotations, activity parsing structures, and co-occurrence relationships. Its dataset includes diverse annotators, images, people, objects, activities, and PaSta annotations, while PaSta-R has multiple implementations.
- Dataset Details: The dataset contains about 118K images, and t-SNE visualizations illustrate part-level features using body-part patches.The visualized examples include “hand holds something” and “foot kicks something.”
- Dataset Details: PaStaNet decomposes the human body into ten parts and assigns descriptive PaSta states to each part.Examples include “hold something” or “push something” for a hand and “watch something” or “eat something” for a head.
- Dataset Details: Approximately 360 annotators with varied backgrounds contributed to PaStaNet, supporting annotation diversity and reducing bias.Their backgrounds are documented in Figure 7.
- Dataset Details: PaStaNet includes person-object interactions and body-only motions, with interacted objects drawn from 80 COCO categories.The dataset lists 156 activities and 76 PaSta categories.
- Knowledge Structures: PaStaNet represents activity knowledge through an activity parsing tree and an activity–PaSta co-occurrence matrix.Parts of both structures are shown in Figures 8 and 9.
- PaSta-R Implementations: PaSta-R implementations include linear, MLP, GCN, sequential LSTM, and tree-structured passing models.These variants receive Activity2Vec outputs in different reasoning architectures.
D. Additional Details of MNIST-Action
The MNIST-Action experiments compare instance-based and hierarchical models for inferring activity labels from composed digit-like parts. The hierarchical approach largely outperforms the instance-based approach.
- Experimental setup: The experiment uses instance-based and hierarchical models to evaluate activity inference from composed digit images.The setup treats digits as body parts and the digit union as the human instance.
- Model comparison: The hierarchical model first recognizes individual digits, then combines part features with instance features to infer the target sum.
- Results: The hierarchical method largely outperforms the instance-based method.
E. Effectiveness on Few-shot Problems
The few-shot analysis groups HICO activities by training-sample count and evaluates performance across these sets. The proposed hierarchical method outperforms the previous state of the art on all sets, especially few-shot sets.
- Evaluation setup: HICO activities are divided into sets according to their numbers of training samples.
- Evaluation setup: Performance has an obvious positive correlation with the number of training samples on HICO.
- Evaluation setup: The few-shot evaluation uses mAP across activity sets, with one-shot performance represented by the set whose training-image count is at most one.
- Results: The proposed hierarchical method outperforms the previous state of the art on all sets, especially few-shot sets.
F. Additional Activity Detection Results
Additional activity-detection results include qualitative PaSta and activity predictions and identify the pre-training datasets used across experiments. The visualizations show predictions spanning interactions with various objects.
- Qualitative results: The method visualizes predicted PaSta triplets and final activity predictions for activity-detection examples.Body parts, verbs, and objects are represented with distinct colors, while correct and incorrect predictions use different markers.
- Qualitative results: The model detects varied activities involving interactions with various objects in the visualized examples.
- Experiment organization: The additional-results tables organize experiments by pre-training data from HICO, HICO-DET, V-COCO, and AVA.
- Experiment organization: Table 8 identifies HICO-DET as the pre-training dataset for the Sec. 5.4 transfer-learning setting.
G. Data usage
The data-usage protocol excludes testing data from all pre-training and fine-tuning procedures to avoid data pollution.
- Data separation: Testing data are carefully excluded from all pre-training and fine-tuning procedures.The stated purpose is to avoid data pollution.