Source-linked AI summary
Uncertainty-aware Score Distribution Learning for Action Quality Assessment
Yansong Tang, Zanlin Ni, Jiahuan Zhou, Danyang Zhang, Jiwen Lu, Ying Wu, Jie Zhou
TL;DR
Existing AQA regression methods overlook ambiguity in scores produced by multiple judges and subjective appraisals. USDL learns score distributions, while MUSDL uses fine-grained scores to model disentangled score components. Experiments on three AQA datasets report state-of-the-art Spearman correlations.
Problem
Existing regression-based AQA methods ignore ambiguity in score labels caused by multiple judges and subjective appraisals.
Method
USDL learns Gaussian score distributions with KL divergence, while MUSDL exploits difficulty degree and judges’ scores as disentangled components.
Results
The approaches achieve state-of-the-art performance under Spearman’s Rank Correlation across three AQA datasets.
Takeaways & Limitations
Fine-grained score labels further improve performance, with MUSDL outperforming the single-path method by 0.4%.
Takeaways & Limitations
Extreme-score videos may receive unsatisfactory predictions because such samples are scarce and the method has difficulty producing extreme outputs.
Abstract
from arXiv · showhide
Assessing action quality from videos has attracted growing attention in recent years. Most existing approaches usually tackle this problem based on regression algorithms, which ignore the intrinsic ambiguity in the score labels caused by multiple judges or their subjective appraisals. To address this issue, we propose an uncertainty-aware score distribution learning (USDL) approach for action quality assessment (AQA). Specifically, we regard an action as an instance associated with a score distribution, which describes the probability of different evaluated scores. Moreover, under the circumstance where fine-grained score labels are available (e.g., difficulty degree of an action or multiple scores from different judges), we further devise a multi-path uncertainty-aware score distributions learning (MUSDL) method to explore the disentangled components of a score. We conduct experiments on three AQA datasets containing various Olympic actions and surgical activities, where our approaches set new state-of-the-arts under the Spearman's Rank Correlation.
1. Introduction
Action quality assessment evaluates performance within the same action category, where poor intra-class discrimination and ambiguous score labels make prediction challenging. The paper addresses this uncertainty by learning score distributions, including disentangled distributions from fine-grained labels.
- AQA evaluates how well a specific action is performed, unlike action recognition, which classifies sequences into categories.
- Regression-based AQA methods predict a single score but disregard ambiguity in labels generated through practical assessment.
- USDL uses a Gaussian score distribution as supervision, with its mean set to the score label, and minimizes KL divergence from predictions.Video features are produced by a 3D ConvNet and used to predict the score distribution.
- MUSDL exploits fine-grained labels such as difficulty degree and judges’ scores to model disentangled components of the final score.The final score is obtained by fusing multiple predicted scores during inference.
2. Related Work
Prior AQA work largely predicts scores through regression, while label distribution learning represents instances with distributions rather than single labels. The paper applies distributional and fine-grained score modeling to AQA.
- Action Quality Assessment: Earlier AQA studies use hand-crafted, convolutional, recurrent, graph-based, and multi-task models to predict action scores.
- Action Quality Assessment: Regression-based AQA methods estimate a single score, whereas USDL predicts a score distribution to represent score uncertainty.
- Label Distribution Learning: Label distribution learning describes each instance with a distribution instead of an original single label or multiple labels.
- Multi-Label Learning: The MTL-AQA dataset provides final scores, action classes, commentary labels, and fine-grained labels usable for multi-label learning.This work uses individual judges’ scores and action difficulty rather than extra labels from other tasks.
- The proposed pipeline segments video frames, extracts I3D features, applies fully connected layers and temporal pooling, then predicts score distributions.The distribution is trained using KL loss against a Gaussian distribution generated from the score label.
3. Approach
The approach learns action-quality score distributions instead of single scores, then extends this framework with multiple paths for judge scores and rule-based aggregation when fine-grained components are available.
- Pipeline Overview: Overlapping video clips are processed by an I3D backbone and shared fully connected layers to produce clip-level features.The clips contain M consecutive frames, and the resulting N features are used by the scoring pipeline.
- Score Distribution Generation: USDL represents each labeled score with a discretized Gaussian score distribution whose uncertainty is controlled by the hyper-parameter σ.The Gaussian is centered at the labeled score, and its values are normalized into the distribution label.
- Learning from Score Distribution: The model maps clip features to predicted score vectors, averages them temporally, applies softmax, and trains with KL divergence against the target distribution.The resulting predicted distribution is spre, and the loss compares spre with pc.
- Inferring from Score Distribution: At inference, USDL selects the score with maximum predicted probability as the final assessment.The final score is obtained from the predicted distribution over discretized scores.
- Learning from Multi-path Score Distributions: MUSDL uses separate sub-networks for multiple judge-score distributions while sharing the I3D backbone across paths.Each path follows the USDL pipeline, with separately trained fully connected layers and K predicted distributions.
- Rule-based Multi-path Inference: During multi-path inference, predicted judge scores are combined according to the game rule, using a subset U and the action’s difficulty degree DD.For diving, the rule discards the two highest and two lowest judge scores; DD may be available or predicted by a side branch.
4. Experiment
Experiments evaluate USDL and MUSDL across Olympic, diving, and surgical action-quality datasets, comparing score-distribution choices, baselines, and fine-grained-score variants. The proposed methods generally improve assessment performance, while extreme-score cases remain difficult.
- Datasets and setup: The evaluation covers AQA-7 sports, MTL-AQA diving, and JIGSAWS surgical tasks, using Spearman correlation and related ranking measures.JIGSAWS includes Suturing, Needle Passing, and Knot Tying, while AQA-7 contains seven sports categories with trampoline excluded.
- Results on AQA-7 Dataset: USDL improves average correlation over the regression baseline by 6.3% and over JRG by 2.5% on AQA-7.USDL significantly improves performance for all action classes except synchronized 10m diving, and exceeds prior methods by about 17% on Snowboard.
- Results on AQA-7 Dataset: Gaussian score distributions achieve the highest average correlation among the tested distributions, while triangle distributions perform worst.The comparison changes only the soft-distribution type; no single distribution performs best for every action class.
- Results on AQA-7 Dataset: Temporal score distributions assign salient low-score predictions to the seventh and eighth clips, where the athlete falls, influencing the final prediction.The visualization uses clip number, score, and predicted probability as its three axes.
- Results on MTL-AQA Dataset: On MTL-AQA, MUSDL outperforms the listed state-of-the-art approaches, while USDLDD exceeds USDL by 1.7% and MUSDL exceeds the single-path method by 0.4%.The ablation attributes these gains to using difficulty degree and fine-grained judge scores through multi-path learning.
- Limitations: The proposed method may perform unsatisfactorily on extreme scores because such samples are scarce and the method is intrinsically less able to produce extreme outputs.In the zero-score case study, both MUSDL and MUSDL* predict scores far above the ground-truth label.
- Results on JIGSAWS Dataset: On JIGSAWS, MUSDL achieves 0.71 (S), 0.69 (NP), 0.70 (Avg. Corr.), and 0.71 (KT), with the best performance on the reported measures except KT being comparable.The experiments use 160 uniformly sampled frames divided into ten segments.
5. Conclusion
The paper introduces USDL to address ambiguity in action score labels and MUSDL to exploit fine-grained score labels. Experiments on three AQA datasets demonstrate the effectiveness of these approaches.
- USDL addresses inherent ambiguity in action score labels by learning from score distributions.
- MUSDL exploits additional fine-grained score labels through a multi-path framework.
- Experiments on three AQA datasets demonstrate the effectiveness of the proposed approaches.
Appendix A. Different Strategies for Segmenting Videos
The appendix compares three video-segmentation strategies and selects 10-seg-s1 for the experiments. This strategy achieves the best result among the evaluated schemes.
- Three strategies were explored for dividing videos into multiple segments.
- The 6-seg scheme normalizes videos to 96 frames and divides them into six 16-frame segments.
- The I3D backbone takes 16 frames as input.
- 10-seg-s1 achieves the best result among the three schemes and is applied to other AQA-7 and MTL-AQA actions.
Appendix B. Visualization of Temporal Evolution
Visualization on Diving shows that the water-entry stage contributes prominently to action-quality assessment. Large splashes correspond to low-score distribution peaks in the affected segments.
- The stage when the player enters the water plays a prominent part in action-quality assessment.
- A large splash from the 8th to the 10th segment corresponds to distributions peaking at low-level scores.