Source-linked AI summary
Learning To Score Olympic Events
Paritosh Parmar, Brendan Tran Morris
TL;DR
Action quality assessment is important but underrepresented by small datasets compared with action recognition. The paper proposes three C3D-based score-regression frameworks and an efficient LSTM training strategy, achieving significant improvements on Olympic-event scoring while retaining LSTM-based feedback capabilities.
Problem
Action quality assessment has received less attention than action recognition and is constrained by datasets typically containing only a few hundred samples.
Method
The paper extracts clip-level C3D spatiotemporal features and predicts scores using SVR, LSTM, or LSTM followed by SVR, with incremental-label training for limited data.
Results
The proposed frameworks significantly improve Olympic-event score prediction over published results, with C3D-SVR performing best while C3D-LSTM-SVR can identify erroneous action segments.
Takeaways & Limitations
SVR-based frameworks provide stronger prediction results, whereas LSTM-based frameworks offer temporal action descriptions that support feedback about where quality suffered.
Takeaways & Limitations
The LSTM feedback mechanism identifies clip-level score gains or losses but does not explain why an error occurred.
Abstract
from arXiv · showhide
Estimating action quality, the process of assigning a "score" to the execution of an action, is crucial in areas such as sports and health care. Unlike action recognition, which has millions of examples to learn from, the action quality datasets that are currently available are small -- typically comprised of only a few hundred samples. This work presents three frameworks for evaluating Olympic sports which utilize spatiotemporal features learned using 3D convolutional neural networks (C3D) and perform score regression with i) SVR, ii) LSTM, and iii) LSTM followed by SVR. An efficient training mechanism for the limited data scenarios is presented for clip-based training with LSTM. The proposed systems show significant improvement over existing quality assessment approaches on the task of predicting scores of Olympic events {diving, vault, figure skating}. While the SVR-based frameworks yield better results, LSTM-based frameworks are more natural for describing an action and can be used for improvement feedback.
1. Introduction
Action quality assessment evaluates how well an action is performed, but unlike action recognition it has received little attention and relies on much smaller datasets. The paper addresses this constraint with visual, spatiotemporal approaches for scoring Olympic sports and introduces datasets, training methods, and feedback capabilities.
- Motivation: Action quality assessment measures how well a person performed an action and can support sports judging and independent exercise-therapy feedback.Automated scoring is also presented as an impartial second opinion in judging contexts.
- Problem: Compared with action recognition, action quality assessment has received little attention and involves subtle differences between action performances.Quality cannot be reliably inferred from only a short segment because errors may occur elsewhere in the action.
- Problem: Whole-action analysis is necessary because a diver can execute the aerial portion well but receive a poor score after an erroneous water entry.A short clip could therefore produce a score that poorly correlates with overall action quality.
- Approach: The paper evaluates visual action-quality frameworks on short-duration diving and vault events and long-duration figure skating.The frameworks are designed for Olympic score prediction under small-dataset constraints.
- Contributions: The work doubles the MIT diving dataset from 159 to 370 samples and introduces a 176-sample gymnastics vault dataset.These datasets support sports score assessment experiments.
- Contributions: The proposed approaches use visual information directly, introduce incremental LSTM training for limited samples, and support locating where action quality deteriorated.Incremental training is reported to reduce training time by about 70%.
2. Related Work
Prior action-quality work largely used human-pose descriptors and addressed sports, surgical skills, or physical therapy. This paper motivates visual spatiotemporal features because pose estimation is challenging and pose-only descriptors can omit scoring-relevant cues.
- Prior work: Only a handful of studies directly address action quality assessment across sports and other application domains.Existing work includes diving, Olympic sporting events, surgical skills, and physical-therapy assessment.
- Pose-based assessment: Earlier Olympic-sports methods estimated human pose frame by frame and transformed concatenated descriptors for SVR-based score prediction.Approximate-entropy features were also used to encode dynamical information.
- Motivation for visual features: Pose estimation is challenging for diving and figure skating because of atypical body positions.This limits the reliability of pose-based representations in these settings.
- Motivation for visual features: Pose-only descriptors can miss scoring cues such as splash size and absolute entry position in diving.These omissions motivate using visual features that capture appearance and temporal evolution.
3. Approach
The approach extracts clip-level spatiotemporal features with C3D and models action quality using SVR, LSTM, or LSTM followed by SVR. Incremental-label training guides LSTMs with intermediate scores for limited-data learning, while temporal score evolution supports clip-level error detection.
- C3D-based feature extraction: C3D extracts spatiotemporal features from video clips for downstream action-quality regression.The method uses C3D representations because action quality depends on both appearance and temporal evolution.
- C3D-SVR: The C3D-SVR variant averages normalized clip features before predicting an action score, sacrificing temporal evolution through aggregation.FC6 activations from 16-frame clips are temporally averaged and supplied to an SVR.
- C3D-LSTM: C3D-LSTM combines sequential clip features with LSTMs that separately encode execution and difficulty scores before regression.C3D features reduce the number of sequence steps and LSTM parameters relative to frame-level processing.
- LSTM final-label training: Final-label training propagates all clips through the LSTM and computes error only after comparing the output with the final event score.This many-to-one setup must infer both partial stage scores and the total score from fewer than 400 examples.
- LSTM incremental-label training: Incremental-label training back-propagates after each clip using an intermediate accumulated score, then applies final-label fine-tuning to relax linear score-growth constraints.Unsupervised intermediate labels evenly divide the total score across clips, while the two-step procedure adapts the model to the final score.
- C3D-LSTM-SVR and feedback: C3D-LSTM-SVR removes the LSTM regression layer and trains an SVR on LSTM activations, combining temporal modeling with shallow-model generalization.The LSTM’s temporal score evolution is also used to identify good and poor action components, although it does not explain why a clip is poor.
4. Experiments & Results
The experiments evaluate C3D-based regression frameworks on judged Olympic sports, with diving receiving the most detailed analysis. C3D-SVR generally performs best, while LSTM-based variants provide temporal score evolution that supports error detection but can have lower correlation.
- Evaluation setup: The frameworks were evaluated on figure skating, diving, and gymnastic vault using Spearman rank correlation for predicted versus true scores.Higher ρ indicates better rank correlation, emphasizing relative ordering rather than exact score values.
- Diving: FC-6 features from the smaller UCF-101-trained C3D network were selected for subsequent evaluations after outperforming the full C3D in the initial diving investigation.The selected small-C3D features also showed 85% zeros, compared with 79% for full-C3D features.
- Diving: C3D-SVR outperformed published diving results, with rank correlation 80% higher than Pose+DCT and 65% better than ApEnFT on the original MIT dive data.Adding known dive difficulty on UNLV-Dive produced a further 0.08 correlation improvement with the optimal temporal stride of four.
- Diving: Incremental-label training achieved ρ = 0.44 for execution score versus ρ = 0.14 for final-label training, while overall score correlation was ρ = 0.36 and below Pose+DCT.Incremental training required 1,000 iterations instead of 10,000 on MIT-Dive; difficulty-score prediction favored final-label training but neither method worked well.
- Diving: The LSTM-SVR variant improved over original C3D-LSTM and approached C3D-SVR performance while retaining clip-level error detection for athlete feedback.Figure 4 illustrates score changes associated with splash size, entry position, and poor body form across dive clips.
- Gymnastic vault: C3D-SVR was best on vault, whereas incremental-label C3D-LSTM achieved only ρ = 0.05 and adding SVR improved performance to almost quadruple Pose+DCT.The vault protocol fixed videos to 100 frames, yielding LSTM sequences of six clips.
- Overall findings: The study concludes that C3D-SVR gives the best results, while LSTM formulations offer interpretability through temporal score evolution despite lower rank correlation.The reported feedback identifies clip-level gains and losses but does not explain why an error occurred spatially or temporally.
5. Conclusion
The paper presents three C3D-based action-quality frameworks for Olympic vault scoring, with different strategies for aggregating clip-level features into video-level descriptions.
- Three frameworks—C3D-SVR, C3D-LSTM, and C3D-LSTM-SVR—improve upon published action-quality assessment results.They mainly differ in how clip-level C3D features are aggregated into a video-level action description.