PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Action Quality Assessment: Method and Benchmark

Kanglei Zhou, Ruizhi Cai, Liyuan Wang, Hubert P. H. Shum, Xiaohui Liang

arXiv:2412.11149v2cs.CV

TL;DR

Heterogeneous datasets and evaluation protocols make systematic comparison across AQA methods difficult. This survey organizes approaches by modality and benchmarks representative video-based methods across datasets and standardized settings. The benchmark finds no single method consistently leads across datasets, and reports an accuracy-computation trade-off.

  • Problem

    AQA surveys and studies use heterogeneous datasets, protocols, and taxonomies, limiting systematic comparison and obscuring shared methodological foundations.

    Source: §1 · p. 2, §6.2 · p. 15

  • Method

    The survey proposes a modality-driven taxonomy and a unified benchmark integrating six datasets and seven evaluation metrics for representative video-based methods.

    Source: §1 · p. 3, §4 · p. 6, §6.2 · p. 15

  • Results

    No method consistently performs best across datasets; dataset characteristics influence performance, and higher predictive accuracy often comes with greater computational cost.

    Source: §6.2.4 · p. 16

  • Takeaways & Limitations

    The survey organizes methodological evolution and identifies reliability and interpretability as near-term directions, alongside generative models and embodied AI as longer-term opportunities.

    Source: §1 · p. 3, §8 · p. 18

    The benchmark covers video-based AQA; skeleton-based and multi-modal methods are excluded because standardized datasets and reproducible pipelines remain limited.

    Source: §8 · p. 18

Abstract

from arXiv · show

Action Quality Assessment (AQA) aims to automatically evaluate how well human actions are performed and has been widely applied in sports analysis, skill assessment, and healthcare. However, AQA studies are often developed under heterogeneous datasets and evaluation settings, making systematic comparison across methods difficult. To address these challenges, we present a comprehensive survey of recent advances in AQA. In particular, we propose a modality-driven hierarchical taxonomy that organizes existing methods into video-based, skeleton-based, and multi-modal approaches, and analyze the methodological evolution of representative models. We further establish a unified benchmark for representative video-based AQA methods by integrating diverse datasets and standardized evaluation protocols, enabling consistent comparison in terms of both accuracy and computational efficiency. Finally, we analyze emerging research trends, identify key challenges in current AQA research, and outline future directions ranging from near-term methodological advances to longer-term opportunities enabled by emerging AI paradigms. The project web page can be found at https://ZhouKanglei.github.io/AQA-Survey.

1. Introduction

AQA evaluates performance differences within the same action category, where subtle motion variations matter, yet existing surveys leave methodological organization and unified comparison underdeveloped. This survey addresses those gaps with a modality-driven taxonomy, a unified benchmark, and analysis of research directions.

1. Introduction

  • AQA assesses performance differences within an action category, requiring sensitivity to subtle motion dynamics and domain-specific evaluation criteria.This distinguishes it from action recognition, which identifies action categories.Source: §1 · p. 2
  • Existing surveys emphasize domain or trend summaries but lack unified methodological organization, systematic benchmarking, and substantial synthesis of future directions.Early surveys use domain-driven taxonomies, while trend-oriented reviews do not unify overlapping techniques across paradigms.Source: §1 · p. 2
  • Heterogeneous datasets and evaluation protocols, narrow validation settings, and diverse application domains fragment AQA research and obscure shared modeling principles.These differences hinder reproducibility and generalization across methods.Source: §1 · p. 3
  • The proposed taxonomy groups methods into video-based, skeleton-based, and multi-modal approaches because input type guides model design.It further distinguishes methods by modality-specific attributes.Source: §1 · p. 3
  • The unified benchmark integrates six datasets and seven evaluation metrics to compare representative methods on accuracy and computational overhead.Standardized experimental settings are intended to support consistent comparisons across diverse methods.Source: §1 · p. 3
  • The survey identifies near-term directions in reliability and interpretability and longer-term opportunities in generative models and embodied AI.It also analyzes emerging trends and challenges in AQA.Source: §1 · p. 3

2. Review Methodology

The review uses a PRISMA-based process to identify, screen, and select AQA studies, then extracts shared attributes for its taxonomy and analysis. It includes 135 methodological papers after applying defined eligibility and quality criteria.

2. Review Methodology

  • The review searches five scholarly sources for publications from 2014 onward and supplements database searches with backward citation analysis.Search terms include action quality assessment, skill assessment, and action scoring.Source: §2 · p. 3
  • Studies must address AQA prediction and offer a relevant learning method, dataset, or evaluation framework, with methodological clarity and benchmark relevance required.The review also excludes non-English, tutorial, abstract-only, duplicate, and non-prediction works.Source: §2 · p. 3
  • After deduplication, title-and-abstract screening and full-text assessment yielded 135 included methodological AQA papers.Papers lacking sufficient technical detail or clear experimental validation were excluded; author disagreements were resolved through discussion.Source: §2 · p. 3, Figure 2 · p. 4
  • For each study, the authors extract input modality, architecture, supervision, datasets, and application domains to support taxonomy and cross-method analysis.The survey also maintains a public webpage of curated papers and metadata, intended for continuous updates.Source: §2 · p. 4

3. Fundamentals of Common AQA Setup

AQA uses a shared framework to turn one or more modalities into predicted quality outputs, while the form of the output and the evaluation metrics depend on the assessment scenario. The section relates continuous scoring, discrete grading or ranking, and task-specific evaluation to different precision, cost, and robustness needs.

3.1. Problem Formulation

  • AQA processes modality-specific features through a backbone, a representation-learning neck, and a prediction head that outputs scores, grades, or ranks.The framework covers inputs from one or multiple modalities.Source: Figure 3 · p. 4, §3.1 · p. 4
  • For unimodal AQA, the model uses one input type; multimodal AQA uses auxiliary modalities for more comprehensive analysis.Source: §3.1 · p. 4

3.2. Typical Scenarios

3.2.3. Implications for Modeling

  • Continuous scoring requires precise, robust labels; grading favors efficiency and interpretability, while ranking reduces annotation cost and emphasizes ordinal consistency.These scenarios trade off precision, cost, and robustness.Source: §3.2.3 · p. 5

3.3.1. Correlation Metrics

  • SRCC measures the strength and direction of the relationship between predicted and true scores by comparing their ranks.It is useful when relative ordering matters more than precise numerical scores.Source: §3.3.1 · p. 5

3.3.2. Precision Metrics

  • rMSE avoids the effect of differing score scales across action categories by normalizing score error and multiplying by 100.The section also identifies MAE and MSE as common measures of continuous-score error.Source: §3.3.2 · p. 5
  • Accuracy measures the proportion of correctly predicted grades or ranked pairs, with an indicator marking matches between predictions and targets.Source: §3.3.2 · p. 5

3.3.3. Task-Specific Metrics

  • IoU evaluates segment localization and average forgetting tracks retention in continual AQA; correlation and precision metrics capture complementary aspects but task-specific metrics reduce cross-task comparability.Ranking-based AQA favors correlation metrics, whereas continuous scoring or grading requires precision metrics.Source: §3.3.3 · p. 5, §3.3.4 · p. 5

4. A Modality-Driven Taxonomy of AQA Methods

The survey organizes AQA methods by input modality because modality shapes motion cues, data representation, and modeling strategy. Its taxonomy separates video-based, skeleton-based, and multimodal approaches, then distinguishes their core challenges and methodological advances.

4. A Modality-Driven Taxonomy of AQA Methods

  • Input modality is the survey’s organizing axis because it determines network design and modeling strategy, while modality-specific trade-offs inform system design.Source: §4 · p. 6

4.1. Modality-Driven Taxonomy: Rationale and Overview

  • The taxonomy divides methods into video, skeleton, and multimodal branches, addressing redundant visual sequences, noisy or costly pose data, and modality fusion, respectively.Source: §4.1 · p. 6
  • The survey distinguishes paradigm-level advances that introduce new modeling formulations from incremental improvements that refine existing pipelines.Source: §4.1 · p. 6

4.2. Video-Based Approach

Video-based AQA methods are organized by three stages: feature extraction, video-level representation learning, and quality prediction. Their designs address computational cost, fine-grained spatial and temporal cues, and the choice between absolute and relative assessment.

4.2. Video-Based Approach

  • Video-based AQA faces long, redundant videos, subtle quality cues, domain-shifted features, and limited labels, motivating designs across three processing stages.The stages are feature extraction, video-level representation learning, and quality prediction.Source: §4.2 · p. 6

4.2.1. Feature Extraction

  • Pretrained 2D CNNs, 3D CNNs, and transformers offer different temporal modeling and computational trade-offs, while AQA's small datasets and long videos make training from scratch or extensive fine-tuning difficult.Recent approaches adapt backbones or refine representations later in the pipeline.Source: §4.2.1 · pp. 6–8

4.2.2. Video-Based Representation Learning

  • Spatial-aware representation learning emphasizes human- and object-centric cues, using spatial priors or refined features to reduce background distraction.Distillation-based refinement transfers spatial knowledge, whereas cue-driven methods add semantic signals but depend on auxiliary cue quality and can add complexity.Source: §4.2.2 · pp. 8–9
  • Temporal-aware methods recover action procedure and long-range dependencies from fragmented clips, using either data-driven sequence models or procedural annotations.Implicit approaches are flexible but can be less interpretable and more costly; explicit approaches provide stage-level feedback but require annotations and may generalize poorly to unseen actions.Source: §4.2.2 · p. 9, Figure 8 · p. 9

4.2.3. Quality Prediction

  • Quality prediction maps video-level representations to outputs through direct assessment of absolute scores or contrastive assessment of relations between actions.Direct methods are efficient but less sensitive to subtle differences; contrastive methods improve fine-grained discrimination and reduce annotation burden at higher computational cost.Source: §4.2.3 · pp. 9–10, Figure 10 · p. 10

4.3. Skeleton-Based Approach

Skeleton-based AQA models joint trajectories rather than visual appearance, and the section emphasizes the distinctive challenges of acquiring and representing skeletal data. Acquisition choices and representation methods trade precision, practicality, and robustness.

4.3. Skeleton-Based Approach

  • Skeleton-based AQA focuses on body kinematics and suppresses background variation, but pose errors, sparse motion cues, and missing environmental context constrain its use.The survey emphasizes these distinctive challenges rather than an exhaustive method taxonomy because relatively few skeleton-based studies exist.Source: §4.3 · p. 10

4.3.1. Skeletal Acquisition Methods

  • RGB cameras are accessible but less precise, depth sensors provide accurate 3D joints with cost and environmental constraints, and MoCap offers high precision in expensive, controlled setups.The appropriate acquisition method depends on balancing accuracy and practicality.Source: §4.3.1 · pp. 10–11, Table 5 · p. 11

4.3.2. Skeletal Representation Learning Methods

  • Skeletal representation learning has shifted from handcrafted descriptors to deep models, with GCNs prominent for modeling joints and their spatial-temporal connections.Robustness mechanisms include hierarchical body-part modeling, complementary inputs and ensembles, auxiliary-task learning, contrastive reasoning, and self-supervision.Source: §4.3.2 · p. 11, Figure 11 · p. 11

4.4. Multi-Modal Approach

Multi-modal AQA combines complementary sources such as audio, text, skeletons, and sensors with visual information. The surveyed methods use these inputs for rhythm and temporal cues, semantic guidance and feedback, or richer kinematic and contextual representations.

Audio-Assisted Methods

  • Audio-assisted methods add rhythmic and temporal cues, progressing from audio-visual modeling in skating and dance to multi-modal fusion, language guidance, and completion of missing modalities.The survey describes these developments as enhancing rhythm awareness, temporal alignment, and robustness.Source: §4.4 · p. 12

Text-Assisted Methods

  • Text-assisted AQA uses language to enrich visual representations with domain semantics or generate interpretable feedback and prompt-guided assessments.The surveyed approaches include text-to-visual knowledge transfer, video-text matching, expert-like feedback, and quality-aware prompts.Source: §4.4 · p. 12

Video-Skeleton Methods

  • Video-skeleton methods fuse visual appearance with explicit kinematics, especially for ambiguous appearance or actions requiring fine-grained body control.Examples include pose-guided transformers and contrastive fusion for multi-person or long-term sports routines.Source: §4.4 · p. 12

Other Modality-Assisted Methods

  • Other methods combine video with wearable, physiological, gaze, optical-flow, or keypoint signals for richer assessment inputs.The surveyed examples include windsurfing, figure skating, cricket, and fitness settings.Source: §4.4 · p. 12

4.5. Modality Selection and Trade-offs

Video, skeleton, and multi-modal AQA differ in information richness, robustness, efficiency, and practical constraints. The survey recommends choosing modalities according to application needs rather than performance alone.

4.5. Modality Selection and Trade-offs

  • Video offers broad visual context but costs more and is sensitive to backgrounds; skeletons are efficient and interpretable but depend on pose quality and may lack context; multi-modal fusion adds robustness but complicates deployment.Multi-modal methods require synchronized acquisition, modality alignment, and full-modality availability at inference.Source: §4.5 · p. 12, Table 6 · p. 13
  • Choose skeleton methods for efficient kinematic analysis, video when visual context matters, and multi-modal designs when robustness justifies annotation and inference complexity.The survey identifies modality-flexible frameworks that selectively activate auxiliary signals as an emerging direction.Source: §4.5 · p. 13

5. Task-Specific Applications

The survey reviews semi-supervised, continual, and interpretable AQA, highlighting label scarcity, adaptation, and useful feedback as practical challenges.

Semi-Supervised AQA

  • Semi-supervised AQA uses unlabeled data with limited labeled samples, but pseudo-label reliability remains an open challenge.Examples include masked segment recovery and teacher–student pseudo-labeling.Source: §5 · p. 13

Continual AQA

  • Continual AQA studies adaptation to new distributions without catastrophic forgetting, while balancing adaptability, memory efficiency, and long-term stability remains open.Representative approaches address domain shift, new action categories, or privacy-aware stabilization.Source: §5 · p. 13

Interpretable AQA

  • Interpretable AQA augments scores with narrative diagnoses or coaching feedback and visual aids, but informative explanations remain challenging.Feedback ranges from keypoint-level analysis to free-form text, with motion overlays and side-by-side comparisons as visual aids.Source: §5 · p. 13

6. Dataset and Benchmark

The survey reviews AQA datasets and benchmarks video-based methods under standardized settings. Results show dataset-dependent performance and trade-offs between predictive accuracy and computational cost.

6.1. Datasets

  • Recent datasets add multi-modal inputs and structured labels, including step-level, subaction, segmentation, and pose-enhanced supervision.Healthcare datasets also include 3D skeletal signals.Source: §6.1 · p. 13
  • Larger datasets now provide thousands to tens of thousands of videos, while LOGO and Trampoline-AQA target minute-level actions.These long sequences pose challenges for temporal modeling and efficiency.Source: §6.1 · p. 13
  • AQA datasets have expanded from sports and healthcare into teaching, AI-generated action evaluation, and traditional activities with specialized criteria.Examples include TAQR and TAQA for teaching, GAIA and Human-AGVQA for AI-generated actions, and BDJ for traditional activities.Source: §6.1 · pp. 13, 15

6.2.1. Experimental Setting

  • The video-based benchmark covers six datasets, seven baseline methods, and seven metrics; it excludes skeleton-based and multi-modal methods because standardized datasets and reproducible implementations remain limited.Datasets span short- and long-term actions; selected baselines represent direct regression, contrastive methods, transformers, and graph reasoning.Source: §6.2 · p. 15, §6.2.1 · p. 15
  • Benchmark results use public implementations, official splits, and three runs without per-dataset or per-method hyperparameter tuning.Experiments use a single RTX 3090 GPU and an I3D backbone unless otherwise specified; task-specific tuning may improve individual methods.Source: §6.2.1 · p. 15

6.2.2. Assessment Performance

  • Transformers perform competitively on several long-sequence datasets, contrastive methods on large-scale datasets, DAE on Fis-V, and HGCN on AQA-7.USDL/MUSDL shows greater variance on challenging datasets such as Fis-V; contrastive-method performance varies more across smaller datasets.Source: §6.2.2 · p. 15

6.2.3. Computational Performance

  • 46 and 49 hours are the respective training times for CoRe and T2CR; CoFInAl has the fastest inference at 0.03s.Contrastive methods reach up to 1.4s per-sample inference, while DAE has the lowest FLOPs at 557.53G and T2CR’s complexity is over 30 times higher.Source: §6.2.3 · pp. 15–16, Figure 12 · p. 15

6.2.4. Experimental Observations

  • No method consistently leads across datasets; dataset temporal length and domain structure influence results, and higher accuracy often carries greater computational cost.The benchmark identifies a need for methods that are both efficient and robust across diverse datasets.Source: Table 8 · p. 16, §6.2.4 · p. 16

7. Trends, Challenges, and Future Directions

The survey traces AQA’s shift toward structure-aware modeling and identifies persistent challenges in data, adaptation, and trust. It outlines future directions from stronger foundation models to generated and embodied action assessment.

7.1. Current Trends

  • AQA is shifting toward fine-grained, interpretable, and multimodal methods, alongside transformers and pre-trained models for long-range modeling and data efficiency.Fine-grained reasoning aligns representations with temporal segments or sub-actions; output-level approaches aim to provide diagnostic and actionable feedback.Source: §7.1 · p. 16

7.2. Under-explored Challenges

  • Open challenges include redundant video inputs versus subtle cues, domain shift, costly fine-grained labels, missing or unreliable modalities, and ethical reliability in sensitive domains.Reliable quality representations from coarse supervision and graceful degradation under incomplete inputs remain unresolved challenges.Source: §7.2 · p. 17

7.3. Future Directions

  • Future directions range from near-term pipeline extensions to opportunities enabled by emerging AI paradigms.The roadmap connects current trends, challenges, and potential directions.Source: §7.3 · p. 17, Figure 13 · p. 17
  • Foundation models are proposed to address label scarcity with transferable representations, though adapting them to AQA requirements remains open.Examples include VideoMAE V2 and VideoMamba for spatiotemporal representation, MotionGPT for skeletal modeling, and Video-LLaMA for video-language understanding.Source: §7.3 · p. 17
  • Future continual learning should address adaptation and generalization to unseen domains, especially ego-view settings, while reducing reliance on labeled updates and stored rehearsal data.Rehearsal may raise privacy concerns and limit real-world applicability.Source: §7.3 · p. 17
  • Robust multi-modal AQA should handle missing or noisy inputs through flexible architectures, uncertainty-aware fusion, and graceful degradation.Current methods often assume complete and reliable modalities.Source: §7.3 · pp. 17–18
  • Trustworthy AQA research should improve robustness, interpretability, and ethical considerations, and explore affective signals for human-centered assessment.Current methods remain vulnerable to adversarial perturbations and insufficient interpretability for practical deployment.Source: §7.3 · p. 18
  • AQA could assess generated motions and actions in embodied learning or robotics, including realism, correctness, and execution quality.The survey proposes exploring integration with generative and embodied models for fine-grained evaluation and potential feedback loops.Source: §7.3 · p. 18

8. Conclusion

The survey organizes AQA research with a modality-driven taxonomy and introduces a video-based benchmark for accuracy and computational efficiency. It also summarizes trends, challenges, and future directions, while noting limits in dataset coverage and benchmark scope.

8. Conclusion

  • The survey reviews AQA advances, proposes a modality-driven taxonomy, and introduces a video-based benchmark, but space limits coverage of some emerging datasets and the benchmark excludes skeleton-based and multi-modal methods.The benchmark emphasizes fair comparison under official settings rather than extensive task-specific tuning.Source: §8 · p. 18

Related papers

Loading 2412.11149v2…