Source-linked AI summary
What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment
Paritosh Parmar, Brendan Tran Morris
TL;DR
The paper asks whether describing an action and its quality can improve action quality assessment beyond single-score learning. It jointly learns spatio-temporal representations for fine-grained action recognition, commentary generation, and AQA scoring, achieving improved generalization and 90.44% correlation with judged scores.
Problem
Existing AQA and skills-assessment methods train on a single final score, which may be insufficient to characterize complicated actions and have limited performance.
Method
The approach jointly optimizes fine-grained action recognition, commentary generation, and AQA scoring to learn shared spatio-temporal representations.
Results
MTL outperforms STL across architectures and training-set sizes, with C3D-AVG-MTL achieving 90.44% correlation with judged scores and new state-of-the-art results.
Takeaways & Limitations
Multitask supervision learns more generalizable representations that better capture action quality than single-task or action-recognition representations.
Takeaways & Limitations
The evaluation centers on diving, although experiments report reasonable transfer to gymnastic vault and the authors discuss broader applicability.
Abstract
from arXiv · showhide
Can performance on the task of action quality assessment (AQA) be improved by exploiting a description of the action and its quality? Current AQA and skills assessment approaches propose to learn features that serve only one task - estimating the final score. In this paper, we propose to learn spatio-temporal features that explain three related tasks - fine-grained action recognition, commentary generation, and estimating the AQA score. A new multitask-AQA dataset, the largest to date, comprising of 1412 diving samples was collected to evaluate our approach (https://github.com/ParitoshParmar/MTL-AQA). We show that our MTL approach outperforms STL approach using two different kinds of architectures: C3D-AVG and MSCADC. The C3D-AVG-MTL approach achieves the new state-of-the-art performance with a rank correlation of 90.44%. Detailed experiments were performed to show that MTL offers better generalization than STL, and representations from action recognition models are not sufficient for the AQA task and instead should be learned.
1. Introduction
The paper asks whether detailed action recognition and commentary can improve action quality assessment beyond training on a single final score. It proposes end-to-end multitask learning and evaluates it with a new 1,412-sample diving dataset, reporting improved performance and state-of-the-art results.
- Existing AQA and skills-assessment systems typically train on a single final score, which may not sufficiently characterize complicated actions.
- The paper proposes end-to-end multitask learning of spatio-temporal representations for action quality scoring, fine-grained action classification, and performance commentary.
- 1,412 diving samples form the first multitask AQA dataset and the largest AQA dataset to date.
- Performance improved as auxiliary tasks were added, and multitask learning outperformed single-task learning across training-set sizes.
- The C3D-AVG-MTL model surpassed existing AQA approaches, while its learned representations outperformed action-recognition representations on unseen actions.
2. Related Work
Prior AQA and skills-assessment methods largely estimate a final score using task-specific visual or learned features, while captioning research treats commentary separately. This paper positions commentary and detailed action classification as complementary supervision for AQA.
- Earlier AQA methods used pose-based features or C3D features to regress final action-quality scores.
- Skills-assessment work applied handcrafted or convolutional features, ranking losses, and spatial or temporal attention across several skills.
- Existing AQA and skills-assessment frameworks were single-task models that produced only the final AQA score.
- Captioning studies used captions or commentary for representation learning, co-training, retrieval, or video description rather than jointly with AQA.
- The new dataset and dive-classification scheme support comparisons involving diverse AQA data and factorized action categories.
3. Multitask AQA Dataset
The multitask AQA dataset contains diverse diving videos with quality scores, factorized dive labels, and expert commentary. Its construction uses televised competition metadata and analyst descriptions to provide supervision for the three tasks.
- The dataset contains 1,412 diving samples, making it the first multitask AQA dataset and the largest AQA dataset to date.
- Samples were collected from 16 international events to provide more variation than datasets drawn from a single main event.
- The collection spans 10m platform and 3m springboard dives, male and female athletes, synchronized pairs and individuals, and multiple views.
- Television graphics provide dive difficulty and descriptions, while judges’ scores after completion provide the AQA labels.
- Retired-athlete commentators supply verbal accounts describing what the athlete performed and which aspects were correct or problematic.
4. Multitask Approach to AQA
The paper formulates AQA as multitask learning, jointly learning shared spatio-temporal representations for scoring, fine-grained action recognition, and commentary generation. Its architectures combine common backbones with task-specific heads and complementary auxiliary supervision to improve AQA generalization.
- Multitask formulation: Related auxiliary tasks constrain the AQA hypothesis space and tend to produce richer representations that improve main-task performance.The shared backbone branches into task-specific heads, and the total loss sums individual task losses.
- Multitask formulation: MTL jointly optimizes AQA scoring, fine-grained action classification, and commentary generation using shared spatio-temporal representations.The main task is AQA, while recognition and commentary provide auxiliary supervision; recognition includes five fine-grained dive sub-tasks.
- Objective functions: AQA uses a combined L1 and L2 regression objective, while action recognition uses cross-entropy and captioning uses negative log likelihood.The overall objective is a weighted sum of the task losses, with α, β, and γ controlling their contributions.
- C3D-AVG: C3D-AVG aggregates clip-level features by averaging them into video-level representations, while captioning receives individual clip-level features.Its C3D backbone extends through the fifth pooling layer, with subsequent task-specific layers acting as decoders.
- MSCADC: MSCADC uses a C3D-based backbone with batch normalization, removed late pooling, dilated convolutions, and multi-scale aggregation to preserve structural information.The architecture downsamples 96-frame actions to 16 frames, reducing parameters and memory while enabling higher spatial resolution.
- Evaluation: MTL outperforms STL across both architectures and improves as more tasks are added, while C3D-AVG performs better and MSCADC is faster with lower memory requirements.Table 4 summarizes STL versus MTL across the architectures and auxiliary-task configurations.
5. Experiments
Experiments show that multitask learning improves AQA across architectures, training-set sizes, and representation evaluations. C3D-AVG-MTL achieves state-of-the-art performance, while learned representations generalize better to unseen actions than action-recognition representations.
- 5.1. Single-task vs. Multi-task approach: MTL approaches outperform STL for both C3D-AVG and MSCADC, with performance improving as additional tasks are incorporated.The comparison includes AQA, detailed action recognition, and commentary generation.
- 5.1. Single-task vs. Multi-task approach: 90.44% correlation with judged scores gives C3D-AVG-MTL new state-of-the-art AQA performance.
- 5.1. Single-task vs. Multi-task approach: C3D-AVG-MTL outperforms S3D on UNLV-Dive, despite the smaller dataset limiting the available training samples.The reported comparison indicates that MTL can learn effectively from fewer training samples.
- 5.1. Single-task vs. Multi-task approach: C3D-AVG-MTL performs best on detailed dive classification, while captioning results are reported without an existing literature baseline.
- 5.1. Single-task vs. Multi-task approach: MTL consistently outperforms STL as training data decreases, with the performance gap widening for fewer samples.
- 5.2. AQA-orientedness of the learned representations: C3D-AVG-MTL learns better intermediate-layer representations for diving than both action-recognition baselines.The evaluation fits linear regressors to activations from all convolutional layers.
- 5.2. AQA-orientedness of the learned representations: For unseen gymnastic vault, C3D-AVG-MTL performs best within-dataset, while MTL representations degrade more gracefully out-of-dataset.Out-of-dataset evaluation is more challenging and lowers performance for all models.
6. Discussion
The discussion presents MTL as a better-generalizing approach for small AQA and skills-assessment datasets, with supervision that can be obtained from existing footage. The recommended C3D-AVG-MTL model reaches 90.44% correlation but remains below human-expert performance.
- 6. Discussion: MTL performs better than STL because it generalizes better, which is especially important when AQA and skills-assessment datasets are small.
- 6. Discussion: Auxiliary-task supervision is scalable because commentary and action-class information is often available in existing footage with minimal extra effort.
- 6. Discussion: 90.44% correlation with judged scores is achieved by C3D-AVG-MTL, compared with 96% human-expert performance.
- 6. Discussion: The approach is presented as general beyond diving, with models trained on diving working reasonably well for gymnastic vault.