Source-linked AI summary
Group-aware Contrastive Regression for Action Quality Assessment
Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, Jie Zhou
TL;DR
Action quality assessment is difficult because videos can differ subtly while their scores vary substantially, and single-video regression does not exploit relationships among videos. CoRe instead learns relative scores through exemplar-based pairwise comparison and uses GART for coarse-to-fine classification plus local regression. It outperforms previous methods across AQA-7, MTL-AQA, and JIGSAWS, establishing new state-of-the-art results on all three benchmarks.
Problem
AQA must predict fine-grained scores despite subtle differences between videos and large score variations, while most existing methods regress scores from single videos.
Method
CoRe regresses relative scores against exemplar videos, and GART decomposes prediction into coarse-to-fine classification and regression within small intervals.
Results
CoRe establishes new state-of-the-art results on AQA-7, MTL-AQA, and JIGSAWS, including Spearman’s Rank Correlation gains from 81.0% to 84.0%, 92.7% to 95.1%, and 70% to 85%, respectively.
Takeaways & Limitations
The experiments support CoRe as a new generic solution for action quality assessment across multiple AQA tasks.
Takeaways & Limitations
Evaluation of some methods on AQA-7 and JIGSAWS was unavailable because those datasets lacked degree-of-difficulty annotations.
Abstract
from arXiv · showhide
Assessing action quality is challenging due to the subtle differences between videos and large variations in scores. Most existing approaches tackle this problem by regressing a quality score from a single video, suffering a lot from the large inter-video score variations. In this paper, we show that the relations among videos can provide important clues for more accurate action quality assessment during both training and inference. Specifically, we reformulate the problem of action quality assessment as regressing the relative scores with reference to another video that has shared attributes (e.g., category and difficulty), instead of learning unreferenced scores. Following this formulation, we propose a new Contrastive Regression (CoRe) framework to learn the relative scores by pair-wise comparison, which highlights the differences between videos and guides the models to learn the key hints for assessment. In order to further exploit the relative information between two videos, we devise a group-aware regression tree to convert the conventional score regression into two easier sub-problems: coarse-to-fine classification and regression in small intervals. To demonstrate the effectiveness of CoRe, we conduct extensive experiments on three mainstream AQA datasets including AQA-7, MTL-AQA and JIGSAWS. Our approach outperforms previous methods by a large margin and establishes new state-of-the-art on all three benchmarks.
1. Introduction
Action quality assessment requires fine-grained score prediction despite subtle differences among videos and substantial score variation. CoRe addresses this by comparing videos through relative-score regression, while GART simplifies prediction into classification and local regression.
- Motivation: AQA predicts fine-grained action-quality scores for videos performing the same action, unlike action recognition and detection tasks.The difficulty arises from subtle inter-video differences and large score variations.
- Challenges: Human-judge subjectivity, subtle differences between similar videos, and limitations of Spearman’s Rank create challenges for accurate AQA evaluation.These challenges affect both score prediction and the interpretation of model performance.
- Contrastive Regression: CoRe regresses relative scores between an input video and exemplar videos instead of directly predicting an unreferenced score.The framework uses differences between videos to guide action-quality assessment.
- Group-Aware Regression Tree: GART divides relative-score prediction into coarse-to-fine classification among intervals and regression within the selected small interval.The framework also introduces R-ℓ2, which accounts for intra-class variance when measuring AQA performance.
- Results: 81.0% to 84.0% on AQA-7, 92.7% to 95.1% on MTL-AQA, and 70% to 85% on JIGSAWS under Spearman’s Rank Correlation demonstrate large gains over previous methods.The experiments cover three mainstream AQA datasets containing Olympic and surgical actions.
2. Related Work
Prior AQA methods generally formulate assessment as regression from video features to final quality scores, building on hand-crafted representations and later deep-learning approaches.
- Existing AQA Methods: Mainstream AQA methods formulate the task as regression using reliable scores provided by expert judges.The related work includes skeleton trajectories, DCT body-pose features, and SVR mappings to final scores.
- Feature-Based Methods: Early approaches use skeleton trajectories or DCT-encoded body pose as input features for action-quality assessment.
- Regression Models: SVR is used to map extracted features to the final action-quality score.
3. Approach
The approach reformulates action-quality assessment as relative-score regression between an input and a comparable exemplar, then predicts that difference with a group-aware regression tree. The tree combines coarse-to-fine group classification with regression inside a smaller score interval, using balanced score-difference partitions and multiple exemplars at inference.
- Contrastive Regression: CoRe compares each input video with an exemplar sharing attributes such as category and difficulty, and regresses their relative score.The paired features are combined with the exemplar’s reference score before predicting the score difference.
- Contrastive Regression: The exemplar-based formulation treats assessment as residual learning by predicting the score difference between the input and reference videos.The approach uses a reference video score rather than directly predicting an unreferenced score from one video.
- Group-Aware Regression Tree: GART divides the relative-score range into 2^d non-overlapping groups and uses a binary tree to classify the difference before interval-specific regression.The leaf layer performs regression within the selected small interval.
- Group-Aware Regression Tree: The tree architecture propagates features top-down, computes leaf probabilities along root-to-leaf paths, and predicts score differences for corresponding groups.Each node produces a binary probability and updated feature, while leaf outputs are mapped to the score-difference range.
- Group-Aware Regression Tree: The proposed partition strategy balances training pairs across score-difference groups instead of uniformly dividing the full range.Uniform partitioning can produce large frequency differences among groups, whereas the proposed grouping balances the pairs.
- Inference: During inference, multiple exemplars are paired with the test video and their predicted scores are combined through multi-exemplar voting.The implementation selects multiple training exemplars to construct separate test pairs.
4. Experiments
Experiments evaluate CoRe and GART across three AQA benchmarks, using Spearman’s rank correlation and R-ℓ2 alongside ablations and analyses of key hyperparameters. The method improves over prior approaches and baselines, with gains reported across AQA-7, MTL-AQA, and JIGSAWS.
- AQA-7 results: CoRe achieves the best results on almost all AQA-7 classes under both Spearman’s correlation and R-ℓ2, with nearly 3.7% and 0.45 average improvements over USDL.The reported class-level Spearman improvements are 8.95%, 2.32%, 8.83%, -6.82%, 3.01%, and 2.25%.
- Hyperparameter analysis: Tree depths of 5 and 6 perform best on AQA-7 Diving, while increasing exemplar count improves performance and lowers variance, with smaller Spearman gains beyond 10 exemplars.The model remains robust across tested tree depths, and exemplar voting increases inference computational cost as the number of exemplars grows.
- MTL-AQA results: On MTL-AQA, CoRe + GART* improves over MUSDL* by 2.0% Spearman and 0.244 R-ℓ2 without difficulty labels, increasing to 2.6% and 0.191 with them.The reported scatter plots and cumulative score curves also show more accurate predictions for CoRe + GART, especially when difficulty information is used.
- Ablation study: Replacing the baseline MLP with GART improves Spearman’s rank by 0.0022 and R-ℓ2 by 0.028, with further gains after adding CoRe.The ablation attributes the improvements to the two components of the proposed method.
- JIGSAWS results: CoRe + GART largely improves prior state-of-the-art results on JIGSAWS and produces more balanced performance across action classes.A four-fold cross-validation protocol is used for this surgical-action dataset.
5. Conclusions
The paper proposes CoRe to learn action-quality differences relative to exemplar videos and combines it with a group-aware regression tree. Experiments on three AQA datasets demonstrate the approach’s effectiveness.
- CoRe learns relative action-quality scores using exemplar videos rather than predicting unreferenced scores.
- The group-aware regression tree converts score regression into coarse-to-fine classification and regression within small intervals.
- Experiments on three AQA datasets demonstrate the effectiveness of the proposed approach.
A. Datasets
The experiments use AQA-7, MTL-AQA, and JIGSAWS, covering Olympic and surgical actions with different annotation and validation setups.
- The study uses AQA-7, MTL-AQA, and JIGSAWS as its three experimental AQA datasets.
- AQA-7 contains 1,189 samples from seven Olympic actions, including diving, vault, and synchronized events.
- MTL-AQA contains 1,412 diving samples from 16 world events with difficulty, judge, action-type, and final-score annotations.
- The experiments adopt the prior protocol for MTL-AQA and a similar four-fold cross-validation strategy for JIGSAWS.
- JIGSAWS contains three surgical tasks, with each video annotated by multiple subscores whose sum forms the final score.
B. More Discussions
Additional analyses examine the regression tree, contrastive regression, and the R-ℓ2 metric. They show how relative comparisons support prediction and provide a stricter, scale-aware evaluation measure.
- More analysis on the regression tree: CoRe + GART performs better than GART at every regression-tree layer for all K values, supporting relative-score comparison over direct prediction.
- More analysis on the regression tree: The regression tree selects the leaf with the highest probability and uses that leaf’s regression value as the final result.
- More analysis on CoRe: CoRe is intended to reduce judge subjectivity by predicting score differences, although exemplar scores are also judge-annotated.
- More analysis on CoRe: Prediction becomes closer to the actual value when more than two exemplars are used, consistent with empirical results in Figure 5(b).
- More analysis on R-ℓ2: R-ℓ2 normalizes score error by an action’s score range and ignores errors below tolerance threshold θ.
- More analysis on R-ℓ2: R-ℓ2 can evaluate individual predictions and is stricter than Spearman’s correlation, which requires the whole test set.
C. Case study
The case study illustrates how the regression tree compares an input video with an exemplar and progressively refines their relative score.
- The first tree layer determines which video is better, while later layers refine that comparison from coarse to fine.
- The case study includes one pair with a small difference and another pair illustrating a larger difference.