Source-linked AI summary
Action Quality Assessment Across Multiple Actions
Paritosh Parmar, Brendan Tran Morris
TL;DR
The paper asks whether shared quality concepts can support action quality assessment across actions. It trains one model on samples from multiple actions and reports improved average Spearman’s rank correlation over action-specific models.
Problem
The paper examines whether actions share quality concepts and whether a single model trained across actions can exploit them for action quality assessment.
Source: §2 · p. 3Method
The authors train a single all-action model on samples from multiple actions rather than separate action-specific models.
Source: §4 · p. 4Results
0.03 average Spearman’s rank correlation improvement over the single-action model; the all-action model performs better on five of six actions.
Source: §5.1 · p. 7Takeaways & Limitations
The authors conclude that multi-action training can leverage limited per-action data more efficiently and provide better initialization for novel actions.
Source: §6 · p. 8The approach is limited to sharing knowledge among tasks from the same domain; cross-domain knowledge sharing was not explored.
Source: §5.4 · p. 8
Abstract
from arXiv · showhide
Can learning to measure the quality of an action help in measuring the quality of other actions? If so, can consolidated samples from multiple actions help improve the performance of current approaches? In this paper, we carry out experiments to see if knowledge transfer is possible in the action quality assessment (AQA) setting. Experiments are carried out on our newly released AQA dataset (http://rtis.oit.unlv.edu/datasets.html) consisting of 1106 action samples from seven actions with quality scores as measured by expert human judges. Our experimental results show that there is utility in learning a single model across multiple actions.
1. Introduction
The introduction argues that small, action-specific AQA datasets may limit model performance and motivates testing whether learning across actions can improve assessment, including for unseen actions. It presents a new AQA dataset and an all-action model as the paper’s response.
1. Introduction
- AQA could support at-home physiotherapy [15], surgical-skills training [5], and sports judging [16]; despite these applications, it has received little research attention.The introduction identifies a resulting dearth of available datasets.Source: §1 · p. 1
- Existing AQA systems train on one action at a time, and expert-dependent data collection leaves datasets small, potentially limiting performance.Source: §1 · p. 1
- The authors ask whether shared action-quality elements can support cross-action training and whether a model trained across actions can assess an unseen action.Source: §1 · p. 1
- The paper introduces a new AQA dataset and an all-action model, and reports evidence that learning across multiple actions has utility over single-action models.The dataset is described as the largest to date.Source: §1 · p. 1
2. Related work
Prior AQA research explored different features, feedback, and skill-assessment tasks, but typically used action-specific models. This work investigates whether a single model can exploit quality concepts shared across actions.
2. Related work
- Pirsiavash et al. used pose-based SVR, Venkataraman et al. reported a 4% improvement with approximate-entropy pose features, and Parmar and Morris [16] explored C3D features.Parmar and Morris compared clip-level averaging and LSTM aggregation, along with SVR and fully connected regression schemes.Source: §2 · pp. 1–2
- Pirsiavash et al. suggested pose tweaks for score improvement, while Parmar and Morris [16] identified potentially erroneous video segments for coaching and faster analysis.Source: §2 · p. 2
- Zia et al. assessed surgical skills using a classifier on frequency-domain transformed spatiotemporal interest points; Doughty et al. [5] assessed surgical skills and everyday tasks.For tasks outside surgery, Doughty et al. used pairwise “which is better” comparisons rather than scores.Source: §2 · p. 2
- Bertasius et al. [2] assessed basketball skills using detected atomic events and Gaussian-mixture features, but annotations reflected a single coach rather than averages across 5–7 judges.The paper contrasts that subjectivity with its scores averaged across individual judges, usually 5–7.Source: §2 · p. 2
- Prior AQA systems used one model per action; drawing on transfer learning [18, 21, 13, 14, 20, 25, 26] and the finding that many classes may aid classification, this work tests a single cross-action AQA model.The motivation is that action-specific models do not exploit shared quality concepts among actions.Source: §2 · pp. 2–3
3. AQA-7 Dataset
AQA-7 introduces samples from seven Olympic actions with consistent scoring rules, addressing data scarcity in action quality assessment. Its sports share quality cues such as difficulty, execution, and landing, though scoring formulas and viewing conditions vary.
3. AQA-7 Dataset
- AQA data collection is difficult because samples require frame boundaries and expert scores, while quality assessment needs the full action sequence.Unlike action recognition, AQA cannot safely use clipped samples or omit frames where execution errors occur, limiting data augmentation.Source: §3 · p. 3
- AQA-7 introduces seven Olympic actions, with scoring rules held consistent across collected samples.The actions are singles diving-10m platform, gymnastic vault, big air skiing, big air snowboarding, synchronous diving-3m springboard, synchronous diving-10m platform, and trampoline.Source: §3 · p. 3
3.1. Description of Actions
- The action set varies in scoring and viewpoint: synchronized diving emphasizes synchronization, Big Air uses four scoring components, and trampoline spans ten phases over 20 seconds.The diving events have negligible view variation, gymvault and Big Air have substantial view variation, and trampoline is filmed from a consistent side view that follows the athlete.Source: §3.1 · p. 3
3.2. Common Action Quality Elements
- Across AQA-7 sports, judges reward difficulty and execution, including twists, spins, straight legs, tight body positions, and clean landings.Final scores combine execution and difficulty differently across sports: diving may use a product, gymvault a sum, and Big Air a more holistic combination.Source: §3.2 · p. 4
4. Our Approach
The approach tests whether shared quality elements enable knowledge transfer by learning one model across multiple actions rather than separate action-specific models.
4. Our Approach
- The all-action model uses frozen C3D fc6 features from 16-frame clips, a 256-dimensional LSTM, and a fully connected score head trained with Euclidean loss.The framework is C3D-LSTM [16], using a smaller C3D network and an LSTM layer; only the LSTM and final fully connected layer are tuned.Source: §4 · p. 4
- The experiments test all-action versus action-specific performance, quality assessment of unseen action classes, and generalization to novel action classes.These three evaluations are designed to test whether quality elements are shared across actions.Source: §4 · p. 4
- Table 2 compares single-action and all-action models using action-wise and average Spearman rank correlation, including feature-averaging/SVR and LSTM/fully connected frameworks.The proposed model is directly comparable to single-action C3D-LSTM [16] because they share the same architecture.Source: Table 2 · p. 5
- Table 3 evaluates zero-shot AQA by training the multi-action model on five action classes and testing on the remaining class, alongside random-initialized and single-action models.Its average correlation uses Fisher’s z-score across the tested classes.Source: Table 3 · p. 5
5. Experiments
Across six short action classes, experiments test joint AQA training, zero-shot transfer, and fine-tuning using Spearman’s rank correlation. Results support multi-action learning as useful for shared quality concepts, with transfer limited to tasks that share action elements.
5. Experiments
- Experiments used 803 training videos and 303 test videos from six similarly short action classes, with action-specific score normalization and six augmented training copies per video.Trampoline was excluded because its videos averaged 650 frames and comprised multiple tricks.Source: §5 · p. 5
5.3. Fine-tuning to a Novel Action Class
- All-action pretraining performed better in 16 of 18 fine-tuning cases and reached high correlation within a few tens of iterations.The advantage also appeared for Gymvault and Skiing, which had shown weak zero-shot results.Source: Table 4 · p. 6, Figure 3 · p. 6, §5.3 · p. 8
5.1. All-Action vs. Single-Action Models
- Across six actions, the all-action model improved average Spearman correlation by 0.03 over single-action C3D-LSTM on five actions, but trailed by 0.14 on Snowboarding.Its 0.6478 correlation was competitive with the action-specific C3D-SVR result of 0.6937.Source: §5.1 · p. 7
5.2.1 Random-initialization vs. Multi-action pre-training
- With five-action training, multi-action pretraining showed positive zero-shot correlation versus near-zero random LSTM initialization on four actions, but Gymvault and Skiing remained weak.The comparison tests whether pretraining learns quality elements that generalize to an unseen action; the random-initialization baseline keeps C3D weights fixed.Source: §5.2.1 · p. 7
5.2.2 Single-Action vs. Multi-Action Transfer
- Single-action transfer was strongest between similar actions and sometimes extended to dissimilar ones, but no single-action model was strong across all classes; all-action training outperformed each on average.The authors attribute the advantage to more shared quality concepts and training datapoints as more actions are included, increasing the chance of overlap with an unseen action.Source: §5.2.2 · pp. 7–8
5.4. Discussion
- The authors conclude that multi-action pretraining is advisable for tasks with shared action elements, while transfer between different domains remains unexplored.Fine-tuning on data already incorporated into the all-action model may overfit; the paper suggests considering related tasks within one domain, such as knot-tying, needle-passing, and suturing.Source: §5.4 · p. 8
6. Conclusion
Experiments on AQA-7 show that training a shared model across multiple actions can improve action quality assessment by leveraging per-action data more efficiently and providing better initialization for novel actions. The findings also suggest potential to extend AQA beyond sports.
6. Conclusion
- A shared model trained across multiple actions leverages limited per-action data more efficiently, improving performance for individual actions.The experiments demonstrate knowledge transfer or sharing in AQA on the AQA-7 dataset.Source: §6 · p. 8
- Multi-action pre-training provides better initialization for novel actions, hinting at an underlying consistency in how quality is judged across actions.Source: §6 · p. 8
- The results suggest that AQA could extend beyond sports scenarios to more general actions.Source: §6 · p. 8