Source-linked AI summary
FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment
Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, Jiwen Lu
TL;DR
Existing AQA methods often regress scores from entire-video deep features, making subtle execution differences difficult to interpret and assess reliably. The paper introduces FineDiving and a procedure-aware Temporal Segmentation Attention method that compares corresponding action steps. The approach achieves substantial improvements over existing AQA methods and state-of-the-art performance on FineDiving.
Problem
Existing AQA methods rely on whole-video deep features, while AQA datasets lack fine-grained procedure annotations and reliable comparisons needed for transparent scoring.
Method
The paper constructs FineDiving and uses Temporal Segmentation Attention to align query and exemplar action steps and perform fine-grained contrastive regression.
Results
The procedure-aware approach achieves substantial improvements and state-of-the-art performance on FineDiving.
Takeaways & Limitations
Fine-grained semantic and temporal procedure annotations support more interpretable action-quality inference.
Takeaways & Limitations
The method assumes that the number of step transitions is known, while fine-grained annotations require manual professional labeling.
Abstract
from arXiv · showhide
Most existing action quality assessment methods rely on the deep features of an entire video to predict the score, which is less reliable due to the non-transparent inference process and poor interpretability. We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. Towards this goal, we construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures. We also propose a procedure-aware approach for action quality assessment, learned by a new Temporal Segmentation Attention module. Specifically, we propose to parse pairwise query and exemplar action instances into consecutive steps with diverse semantic and temporal correspondences. The procedure-aware cross-attention is proposed to learn embeddings between query and exemplar steps to discover their semantic, spatial, and temporal correspondences, and further serve for fine-grained contrastive regression to derive a reliable scoring mechanism. Extensive experiments demonstrate that our approach achieves substantial improvements over state-of-the-art methods with better interpretability. The dataset and code are available at \url{https://github.com/xujinglin/FineDiving}.
1. Introduction
The paper argues that action quality assessment needs fine-grained procedural understanding because whole-video regression is difficult to interpret and may miss subtle execution differences. It introduces FineDiving and a procedure-aware Temporal Segmentation Attention approach for more accurate, interpretable scoring.
- Action Quality Assessment estimates how well an action is performed by analyzing the complete action sequence and predicting a score.
- Whole-video deep-feature regression is unreliable for diving because similar routines and backgrounds contain subtle differences in twists, somersaults, positions, and execution quality.
- Existing AQA datasets lack fine-grained procedure annotations and reliable comparisons, limiting transparent reasoning from final action-quality scores.
- FineDiving provides two-level semantic labels, two-level temporal annotations, and official, judges’, and difficulty scores collected from FINA.
- The Temporal Segmentation Attention module parses query and exemplar actions into corresponding steps, learns procedure-aware cross-attention embeddings, and performs fine-grained contrastive regression.
- Extensive experiments report substantial improvements and state-of-the-art performance for the procedure-aware approach.
2. Related Work
Prior sports-video research developed datasets and action-quality methods, but FineDiving targets fine-grained procedural understanding for AQA. Its semantic and temporal structures organize diving actions into reusable sub-actions and consecutive procedural steps.
- Sports Video Datasets: Sports video understanding is challenging because sports datasets exhibit low inter-class variance in motions and environments.
- Sports Video Datasets: Existing datasets span Olympic sports, large-scale sports recognition, judging, action quality assessment, fine-grained diving attributes, and dense spatio-temporal annotations.
- Two-level Semantic Structure: Its semantic hierarchy represents action routines through combinations of sub-action types, allowing different action types to share sub-actions.
- Sports Video Datasets: FineDiving differs by providing fine-grained annotations specifically for action quality assessment, including procedural semantics and temporal structure.
- Two-level Temporal Structure: Its temporal hierarchy marks complete action boundaries and the starting frames of consecutive procedural steps.
3. The FineDiving Dataset
FineDiving is constructed as a fine-grained diving dataset organized by semantic and temporal procedure annotations. It combines professional labeling, action scores, and broad coverage of action, sub-action, and difficulty categories.
- Dataset Construction: FineDiving is a fine-grained competitive-sports video dataset designed to support dataset construction and statistical analysis for diving AQA.
- Collection: The videos are collected from Olympic, World Cup, World Championships, and European Aquatics Championships events using high-resolution competition footage.
- Lexicon: The dataset uses a two-level semantic lexicon in which sequential sub-action types combine to form an action type.
- Temporal Structure: Temporal annotation records complete action boundaries and starting frames for consecutive steps, while incomplete actions and slow playbacks are discarded.
- Annotation: Professional and cross-validated annotation procedures label sub-action types and step starts, with results checked by another worker.
- Dataset Statistics: 3000 video samples cover 52 action types, 29 sub-action types, and 23 difficulty-degree types, alongside scores and fine-grained semantic and temporal annotations.
4. Approach
The approach uses Temporal Segmentation Attention to assess action quality from pairwise query and exemplar videos by segmenting procedures, aligning corresponding steps, and regressing fine-grained score differences. Its training combines transition prediction with score supervision, while testing uses multiple exemplars for voting.
- Overall Framework: The overall framework combines the I3D backbone with TSA, jointly optimizing transition classification and score regression for pairwise training examples.During testing, the method forms multiple test–exemplar pairs using a multi-exemplar voting strategy.
- Procedure Segmentation: Temporal Segmentation Attention parses query and exemplar actions into consecutive steps by predicting transitions between sub-actions.The segmentation component predicts ordered transition probabilities across frames and identifies each transition at the frame with greatest probability.
- Procedure Segmentation: The segmentation module is optimized with binary cross-entropy against binary transition labels, encouraging predicted transition distributions to match the ground truth.The transition constraint also enforces ordered predictions, ˆt1 ≤ ··· ≤ ˆtL.
- Procedure-aware Cross-Attention: After segmentation, transformer cross-attention learns procedure-aware embeddings between corresponding query and exemplar steps, discovering their spatial and temporal correspondences.Step features are resized when their lengths differ, and pairwise steps guide attention toward consistent exemplar regions while preserving spatial information.
- Fine-grained Contrastive Regression: Fine-grained contrastive regression quantifies deviations between aligned query and exemplar steps and predicts the query score using the exemplar’s labeled score.The regression component is optimized with mean squared error between the query’s ground-truth and predicted scores.
5. Experiments
Experiments evaluate procedure segmentation and action quality assessment using segmentation metrics, rank correlation, and relative score distance. FineDiving comparisons show state-of-the-art performance, interpretable cross-attention, and a performance–computation trade-off as exemplar count increases.
- Evaluation Metrics: AIoU@d evaluates whether predicted step-transition intervals overlap ground-truth intervals beyond threshold d; higher values indicate better procedure segmentation.
- Evaluation Metrics: Spearman’s rank correlation measures AQA performance by comparing predicted and ground-truth rankings, with higher ρ indicating better performance.
- Evaluation Metrics: Relative ℓ2-distance measures score-prediction error relative to the action’s highest and lowest scores, with lower values indicating better performance.
- Results and Analysis: The proposed approach achieves state-of-the-art FineDiving results, improving Spearman’s rank correlation over USDL, MUSDL, and CoRe without dive-number exemplar selection by 6.23%, 4.98%, and 2.94%.It also improves Relative ℓ2-distance over those methods by 0.1145, 0.0951, and 0.0783, respectively.
- Results and Analysis: Increasing the number of voting exemplars improves performance but increases computational cost, with Spearman improvements becoming less significant beyond M >10.Relative ℓ2-distance follows a similar trend.
- Visualization: Procedure-aware cross-attention highlights semantic, spatial, and temporal correspondences between query and exemplar steps, making fine-grained relative-score learning more interpretable.
6. Conclusion and Discussion
The paper presents FineDiving and a procedure-aware AQA approach that learns consistent regions across paired procedures for more interpretable inference. It also identifies known transition counts and professional annotation as limitations, while the dataset uses online diving videos and consent outreach.
- FineDiving is presented as the first fine-grained sports video dataset for assessing action quality.
- The method assumes that the number of step transitions is known, and its fine-grained annotations require manual decomposition and professional labeling.
- The diving dataset is collected from YouTube and bilibili, and the authors are contacting creators to ensure appropriate consent.
A. More Experiment Settings
The experiments select exemplars according to action type during training and inference, with inference using multi-exemplar voting for the main TSA setting.
- For TSA with dive-number selection, training samples one exemplar from remaining training instances sharing the same action type.
- At inference, TSA with dive-number selection randomly selects 10 same-action-type exemplars, whereas TSA without dive-number selection samples exemplars randomly.
A.2. The Number of Step Transitions
FineDiving action procedures contain three to five steps depending on action type, and the dataset annotates five possible steps for future fine-grained AQA research.
- In FineDiving, 60% of action types contain three steps, 30% contain four steps, and 10% contain five steps.
- The dataset provides full annotations for five possible steps, while experiments keep the step-transition count L constant at 2.
B. More Ablation Study
The ablations examine how step length and transformer depth affect procedure-aware action quality assessment, identifying settings that balance information use and overfitting.
- Number of frames in each step: 5 frames per step improves Spearman’s rank correlation by 1.22% and Relative ℓ2-distance by 0.0583 over 3 frames.The reported gains occur as Lstep increases from 3 to 5.
- Number of frames in each step: Performance is not proportional to step length because short steps miss intra-step information while long steps may introduce noise.Performance becomes flat or slightly decreases when Lstep exceeds 8.
- Transformer decoder layers: With 5 frames per step, 3 transformer decoder layers achieve Spearman’s rank correlation of 0.9203 and Relative ℓ2-distance of 0.3420.The experiment compares decoder depths of 1, 3, 5, and 10 layers.
- Transformer decoder layers: More than 5 decoder layers produce flat or slightly declining performance, attributed to overfitting within five-frame steps.The model uses a transformer decoder with 3 layers and 8 heads in the summarized training configuration.
D. The FineDiving Dataset Details
The paper states that the FineDiving dataset and approach code are provided, and that the dataset will be released to promote future action quality assessment research.
- The authors plan to release FineDiving to promote future research on action quality assessment.
D.1. Descriptions of Action and Sub-action Types
FineDiving represents action types through combinations of phase-specific sub-action labels, including variable flight annotations for twists and frame-wise temporal labels.
- A combination of sub-action types from take-off, flight, and entry phases generates an action type.The dataset’s action lexicon describes these action and sub-action relationships.
- Take-off uses one of six directional or armstand sub-action labels, while entry uses the single Entry label.The listed take-off labels are Forward, Back, Reverse, Inward, Armstand Forward, Armstand Back, and Armstand Reverse.
- Flight uses two sub-action labels when twists occur: one for twist turns and one for somersault turns.The twist label is part of the somersault process and is not performed independently.
- Twist location determines whether flight labels are ordered as twist-then-somersault or somersault-twist-somersault.The latter ordering separates the first and last portions of the same somersault process around the twist.
- Without a twist, flight receives one sub-action label representing the number of somersault turns.
- Frame-wise sub-action durations vary across instances, so a public annotation toolbox is used to generate labels efficiently, accurately, and consistently.