Source-linked AI summary
Action Quality Assessment Across Multiple Actions
Paritosh Parmar, Brendan Tran Morris
TL;DR
The paper examines whether action-quality knowledge transfers across actions and whether pooled samples can improve AQA. It trains and evaluates shared multi-action models on AQA-7, finding utility in learning across actions. The authors conclude that multi-action learning improves data use and initialization for novel actions, while transfer is scoped to related domains.
Problem
AQA datasets are small and existing systems usually train action-specific models, leaving shared quality concepts and cross-action transfer insufficiently explored.
Method
The paper trains a single AQA model on samples from multiple actions and evaluates multi-action learning and pre-training on the AQA-7 dataset.
Results
Multi-action models outperform single-action models on five of six actions, improving average Spearman’s rank correlation by 0.03 without changing the network.
Takeaways & Limitations
Multi-action training better leverages limited per-action data and provides improved initialization for novel actions, indicating shared quality concepts across related actions.
Takeaways & Limitations
The approach is intended for actions from the same domain; knowledge sharing across different domains was not explored.
Abstract
from arXiv · showhide
Can learning to measure the quality of an action help in measuring the quality of other actions? If so, can consolidated samples from multiple actions help improve the performance of current approaches? In this paper, we carry out experiments to see if knowledge transfer is possible in the action quality assessment (AQA) setting. Experiments are carried out on our newly released AQA dataset (http://rtis.oit.unlv.edu/datasets.html) consisting of 1106 action samples from seven actions with quality scores as measured by expert human judges. Our experimental results show that there is utility in learning a single model across multiple actions.
1. Introduction
The paper asks whether action-quality knowledge transfers across actions and whether pooled multi-action data can improve AQA. It introduces an all-action modeling approach and reports evidence supporting its utility.
- Automatic AQA quantifies how well an action was performed and could support physiotherapy, medical training, and sports judging.
- Existing AQA systems usually learn one model per action, but expert annotation makes datasets small and may limit performance.
- The paper investigates whether actions share quality elements and whether transfer learning across actions can address limited target data.
- The authors introduce a multi-action model and evaluate whether learning from all actions is useful compared with single-action modeling.
2. Related work
Prior AQA work largely builds separate models for each action, while datasets and assessment schemes remain limited. This paper frames shared quality concepts and pooled learning as alternatives.
- Earlier AQA studies used pose or deep features to predict action quality, with one model for each action.
- AQA research also includes feedback mechanisms that suggest pose adjustments or identify video segments containing possible errors.
- Surgical and general skill-assessment studies extend AQA beyond sports, but some evaluate pairwise preferences or task-specific characteristics.
- Single-action models do not exploit shared quality concepts, motivating evaluation of one model trained on datapoints from all actions.
3. AQA-7 Dataset
AQA-7 addresses limited AQA data with seven Olympic actions sharing movement and scoring elements. Its design preserves full action sequences because execution errors can occur anywhere.
- AQA data collection is difficult because expert scoring is required, and full action sequences must be retained to capture execution errors.
- AQA-7 contains samples from seven Olympic actions, including diving, gymnastic vault, skiing, snowboarding, and trampoline.
- The dataset includes multiple diving events, with synchronized dives additionally assessing athlete synchronization.
- 3.2. Common Action Quality Elements: Across AQA-7 sports, flipping, twisting, body position, landing, execution quality, and difficulty contribute to judged scores.
- 3.2. Common Action Quality Elements: Different sports combine execution quality and difficulty differently, using product, summation, or more holistic scoring functions.
4. Our Approach
The paper tests whether action-quality knowledge can be shared across actions using a C3D-LSTM framework and experiments comparing multi-action, single-action, and unseen-action settings.
- Motivation: The authors hypothesize that different actions share quality elements that can support knowledge transfer despite action-specific judging criteria.They acknowledge that actions may instead be unique, which would prevent consistent cross-action quality modeling.
- Model: The proposed C3D-LSTM processes 16-frame video clips with a frozen C3D feature extractor, a 256-dimensional LSTM, and a fully connected layer producing the AQA score.Only the LSTM and final fully connected layer are tuned, using Euclidean distance between predicted and true scores as the loss.
- Experiments: Three experiments evaluate all-action learning, quality estimation for unseen actions, and generalization to novel action classes.These experiments are designed to test both shared quality elements and transfer across actions.
- Evaluation: Table 2 compares single-action and all-action models using action-wise and average Spearman’s rank correlation, with higher values indicating better performance.The all-action and single-action C3D-LSTM models share the same architecture, enabling direct comparison.
- Evaluation: Table 3 evaluates randomly initialized, single-action, and multi-action models when testing on unseen action classes.The multi-action model is trained on five actions and evaluated on the remaining action, while single-action rows include same-action diagonal results.
5. Experiments
The experiments compare joint multi-action AQA models with action-specific or randomly initialized models, testing transfer to unseen actions and fine-tuning efficiency. Across these settings, multi-action training generally improves correlation, initialization, convergence, and scalability, while transfer depends on shared action elements and domain similarity.
- Experimental setup: The experiments use Spearman’s rank correlation and aggregate results with Fisher’s z-value, evaluating joint versus individual AQA models.The data preparation normalizes action-specific score ranges; experiments use six similar short action classes, with 803 training and 303 test videos.
- All-Action vs. Single-Action Models: The all-action model’s 0.6478 correlation is competitive with action-specific C3D-SVR’s 0.6937 while retaining LSTM-based temporal aggregation.The paper notes that LSTM aggregation is preferred for temporally localizing action-quality drops.
- Random-initialization vs. Multi-action pre-training: On unseen actions, random initialization produces Spearman correlation near zero, whereas multi-action initialization produces positive correlation.The comparison trains on five actions and tests on an unseen sixth action, supporting shared quality elements as the proposed explanation.
- Single-Action vs. Multi-Action Transfer: Single-action models transfer best to similar actions, but cross-action correlations also support shared quality concepts beyond closely matched judging criteria.The Skiing model retains significant rank correlation when tested on Diving despite their different judging criteria.
- Single-Action vs. Multi-Action Transfer: The all-action model greatly outperforms single-action models on average because additional actions provide more datapoints for shared quality elements.Adding actions also reduces the ratio of quality concepts to total datapoints, improving the opportunity to learn those concepts.
- Fine-tuning to a Novel Action Class: Fine-tuning from multi-action pre-training usually starts higher and converges faster than fine-tuning from random initialization.The all-action model reaches high values after a few tens of iterations, and Table 4 reports the best result in 16 of 18 cases.
- Fine-tuning to a Novel Action Class: Even Gymvault and Skiing, which had poor initial zero-shot performance, quickly improve during fine-tuning.The authors suggest local optimization behavior and the relatively high initial learning rate may contribute, while action-specific hyperparameter tuning remains outside scope.
6. Conclusion
The work demonstrates that AQA benefits from knowledge transfer through a shared model trained across multiple actions. Experiments on AQA-7 show improved per-action data use and better initialization for novel actions.
- Training a shared model across multiple actions enables knowledge transfer in action quality assessment.
- Considering multiple actions leverages limited per-action data more efficiently for improved per-action performance.
- Multi-action pre-training provides better initialization for novel actions and suggests consistency in the notion of action quality.