PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to score the figure skating sports videos

Chengming Xu, Yanwei Fu, Bing Zhang, Zitian Chen, Yu-Gang Jiang, Xiangyang Xue

arXiv:1802.02774v3cs.MMcs.CV

TL;DR

The paper addresses learning to score figure-skating videos, where scores depend on understanding long performances and weakly supervised final TES and PCS labels. It proposes complementary Self-Attentive LSTM and Multi-scale Convolutional Skip LSTM models and introduces the Fis-V dataset; experiments on Fis-V and MIT-skate validate the proposed methods.

  • Problem

    Figure-skating scoring requires predicting final TES and PCS from whole, variable-length performances whose clip-level score contributions are unavailable.

  • Method

    The paper combines Self-Attentive LSTM for local information with Multi-scale Convolutional Skip LSTM for multi-scale local and global sequential information, using clip features from C3D.

  • Results

    Experiments on MIT-skate and Fis-V show that the integrated framework and its components outperform the compared baselines on reported correlation and mean-square-error evaluations.

  • Takeaways & Limitations

    Fis-V and the proposed models provide a benchmark and architectures for learning to score professional figure-skating performances.

Abstract

from arXiv · show

This paper targets at learning to score the figure skating sports videos. To address this task, we propose a deep architecture that includes two complementary components, i.e., Self-Attentive LSTM and Multi-scale Convolutional Skip LSTM. These two components can efficiently learn the local and global sequential information in each video. Furthermore, we present a large-scale figure skating sports video dataset -- FisV dataset. This dataset includes 500 figure skating videos with the average length of 2 minutes and 50 seconds. Each video is annotated by two scores of nine different referees, i.e., Total Element Score(TES) and Total Program Component Score (PCS). Our proposed model is validated on FisV and MIT-skate datasets. The experimental results show the effectiveness of our models in learning to score the figure skating videos.

I. INTRODUCTION

The paper frames figure-skating scoring as a whole-video regression problem and proposes complementary recurrent architectures alongside the Fis-V dataset. The dataset contains professionally captured performances with expert scores, while the models target local and global sequential information.

  • Problem: Figure-skating scoring requires understanding entire videos, including approximately 4,400 frames, rather than only recognizing short action segments.A localized mistake can substantially affect the final score, and video lengths vary.
  • Problem: The task is challenging because performances are long, scores come from referees, and only technically or artistically relevant segments contribute to TES or PCS.These properties distinguish scoring from conventional crowdsourced action-recognition labels.
  • Method: The proposed framework combines Self-Attentive LSTM for local information with Multi-scale Convolutional Skip LSTM for local and global multi-scale information.The skip mechanism reduces computation, and either subnetwork can operate alone or within the integrated regressor.
  • Evaluation: The framework is evaluated on both MIT-skate and Fis-V, and the experiments validate the effectiveness of the proposed models.The paper positions the dataset and models as resources for learning to score professional sports videos.
  • Dataset: Fis-V contributes professionally captured, score-labeled figure-skating videos focused on individual performances rather than irrelevant surrounding footage.The videos retain the performance and remove activities such as warming up or post-performance bowing.
  • Dataset: Fis-V contains 500 videos from 149 professional skaters across more than 20 countries, with videos averaging about 2 minutes and 50 seconds.Scores were gathered from nine international referees.

II. RELATED WORK

Prior video research developed spatial-temporal representations, recurrent models, two-stream architectures, and 3D convolutional features. These advances improved video understanding but also exposed computational and representation challenges relevant to scoring.

  • Video representations: Deep video research combines convolutional networks, LSTMs, 3D convolutions, and two-stream models to learn spatial-temporal representations.The cited approaches target visual recognition and action classification.
  • Video representations: Handcrafted trajectory features such as HOF and MBH capture local motion, while deep learning learns task-tailored spatial-temporal representations directly from data.The passage contrasts traditional motion descriptors with learned representations.
  • Computational challenges: 3D convolutions can underperform handcrafted features without large-scale training data and are computationally expensive.The related work motivates methods that preserve temporal information more efficiently.
  • Computational challenges: Factorized spatial-temporal convolutions and two-stream networks were proposed to reduce computational burden and separately learn spatial and temporal representations.Two-stream variants subsequently explored alternative fusion strategies.
  • Feature extractors: C3D provides compact video representations through stacked 3 × 3 × 3 convolutions and is described as preserving temporal information better than 2D CNNs.SENet instead uses Squeeze-and-Excitation blocks to integrate channel-set features.

B. Video Fusion

The section situates sports-video scoring within broader video fusion and sports-analysis research. Earlier work studied multimodal fusion and sports actions, while only a small number of studies addressed score prediction directly.

  • Feature fusion: Video systems commonly use early or late feature fusion, with later methods learning fusion weights or reducing noise through shared low-rank structure.The cited literature includes multiple-kernel and low-rank fusion approaches.
  • Feature fusion: Neural multimodal systems fuse information from modalities such as frames, optical flow, and audio spectrograms using deep architectures or recurrent networks.These approaches extend fusion beyond simple feature concatenation.
  • Sports analysis: Sports-video analysis has examined automated assessment, player trajectories, team behavior, and activity recognition across several sports.The cited examples include gymnastics and basketball applications.
  • Score prediction: Learning to score sports was comparatively less studied, with prior work using pose-based regression and SVR or LSTM models on C3D features.The prior studies included diving and figure skating datasets.
  • Dataset contribution: The Fis-V dataset is designed for score prediction and related figure-skating video-analysis tasks such as shot-highlight generation.The dataset is intended for community release under a necessary license.

A. Dataset construction

Fis-V is constructed from high-level international competitions with standardized scope, carefully trimmed performance footage, and TES and PCS annotations. The paper also analyzes relationships between these scores across matches and skaters and compares the dataset with MIT-skate.

  • Data source: Fis-V draws on formal high-standard international competitions, including NHK Trophy, Trophee Eric Bompard, Cup of China, and Four Continents Championships.The source videos focus on competition performances rather than unrelated material.
  • Selection criteria: The dataset restricts videos to top-level competitions and ladies’ singles short programs to improve score comparability across seasons and genders.The selected competitions include ISU Championships, the ISU Grand Prix, and Winter Olympic Games.
  • Pre-processing: Redundant footage such as warm-ups, bows, waiting, and replays is removed so videos represent the skater’s performance itself.Each video tracks approximately one player throughout the performance.
  • Scoring: Each video is labeled with Total Element Score (TES) and Total Program Component Score (PCS), measuring technical execution and program-performance components.The scores are assigned under the competition mark scheme.
  • Data analysis: Spearman and Kendall tau correlations are computed between TES and PCS across competitions and across individual skaters.The match analysis uses all skaters in each match, while the player analysis uses the same skater across her matches.
  • Data analysis: In more than half of the matches, TES has little correlation with PCS, reflecting that the scores measure different performance perspectives.A few matches show high correlation, which the authors attribute to referee subjectivity.
  • Dataset comparison: Compared with MIT-skate, Fis-V has more than three times as many videos, both PCS and TES annotations, and newer videos from 12 competitions between 2012 and 2017.The comparison emphasizes scale, annotation breadth, and recency.

IV. METHODOLOGY

The framework formulates figure-skating scoring as weakly labeled regression from video-level features to TES and PCS. It uses C3D clip features and organizes the methodology around video representation and scoring fusion.

  • Framework: The methodology obtains video-level representations before applying the video fusion scheme for score prediction.The framework section separates problem setup and features, video-level representation, and video fusion for learning to score.
  • Problem Setup: The task predicts TES and PCS as two independent regression targets from final competition scores.Incremental scores are not synchronized with individual video clips, so only final TES and PCS are provided.
  • Video Features: Video clips are represented with 4096-dimensional fc6 features from a C3D network pretrained on Sports-1M.A 16-frame sliding window with stride 8 extracts the clips.
  • Framework: The multi-scale convolution aggregation model uses skip-LSTM within the video-scoring framework.The model overview is presented as the multi-scale convolution aggregation model with skip-LSTM.

B. Self-Attentive LSTM (S-LSTM)

The Self-Attentive LSTM compresses long C3D feature sequences by emphasizing clips relevant to difficult technical movements. Its compact embedding is processed by an LSTM and regression layer, with a diversity penalty encouraging varied attention patterns.

  • Self-Attentive Embedding: Self-attentive embedding selectively weights clip features because clips do not contribute equally to final score regression.The design targets important clips associated with difficult technical movements while producing fixed-length representations.
  • Self-Attentive Embedding: The attention weights are normalized with softmax, whose weights sum to 1.The embedding uses softmax and hyperbolic tangent functions in its attention computation.
  • Self-Attentive Embedding: A two-layer MLP produces an attention matrix whose rows represent distinct video focus points, such as key action patterns.The compact representation is computed as M = A · F, with d1 describing diversity of descriptions.
  • Self-Attentive Embedding: Multiplying the feature matrix by the attention matrix yields a shorter d2 × d representation sequence.This reduces the sequence supplied to later processing while retaining multiple learned patterns.
  • Regression: The compact embedding feeds an LSTM and fully connected regression layer, while a penalty added to MSE encourages diverse embeddings.The self-attentive LSTM was introduced here for regression and differs from prior attention approaches in how its embedding is used.

C. Multi-scale Convolutional Skip LSTM (M-LSTM)

M-LSTM combines multi-scale convolution with a revised skip LSTM to model local and global sequential information while discarding redundant updates. Its parallel recurrent outputs are concatenated for score regression.

  • Multi-scale representation: M-LSTM uses parallel 1D convolutions with different kernel sizes to extract local and global representations at multiple temporal scales.Small kernels capture shorter action patterns, while larger kernels capture longer-duration patterns.
  • Skip LSTM: A binary update gate controls whether the LSTM updates its cell and hidden states, allowing the model to skip less significant information.The gate takes values 0 or 1 and controls state updates.
  • Skip LSTM: The revised skip LSTM changes the state-update rule to avoid exposing an unupdated memory cell that could produce misleading information.The revision differs from the referenced skip-LSTM formulation and is illustrated in Fig. 5.
  • Skip LSTM: The round-function gate lets M-LSTM skip less significant updates when ut = 0, supporting longer-term data dependencies.This mechanism discards redundant information during sequential modeling.
  • Regression: Outputs from the final time step of the parallel LSTMs are concatenated and passed to a fully connected layer for score regression.Skip LSTM is attached to small-kernel convolution layers, while common LSTM layers follow the other convolution layers.

A. Settings and Evaluation

The evaluation uses MIT-skate and Fis-V with Spearman correlation and MSE, alongside feature, pooling, regression, and recurrent-model competitors. Fis-V is split into 400 training and 100 testing videos, while the reported tables summarize correlation and error results.

  • Datasets: MIT-skate contains 150 videos split into 100 training and 50 testing videos, while Fis-V uses 400 training and 100 testing videos.MIT-skate videos contain 24 frames per second.
  • Metrics: Evaluation uses Spearman correlation ρ for comparability with prior work and MSE for additional model assessment.On MIT-skate, evaluation uses the published final scores.
  • Implementation: The experiments train both subnetworks with Adam at learning rate 1e-4, using batch size 32 and a 256-unit hidden size for the recurrent layers.Training uses one NVIDIA 1080Ti GPU, converges by 250 epochs, and takes about 20 minutes per model.
  • Baselines: The competitor set includes SENet frame features, max or average pooling, linear or RBF SVR, LSTM and bidirectional LSTM models, and prior methods.The C3D-LSTM comparison uses a 256-unit hidden size and a multi-layer regressor matching the proposed models.
  • Reported results: Table I reports Spearman correlation on MIT-skate and Fis-V, while Table II reports MSE on Fis-V.The table caption states that higher Spearman correlation is better and defines S-LSTM and M-LSTM abbreviations.

B. Results

The proposed framework achieves the best Spearman-correlation performance on both MIT-skate and Fis-V, while ablations show complementary and task-dependent contributions from its components. Additional analyses indicate that pooling, feature type, and target score affect performance.

  • Spearman correlation: The framework achieves the best Spearman-correlation performance on both MIT-skate and Fis-V, outperforming the baselines by a large margin.M-LSTM alone beats all other baselines on both datasets, while S-LSTM further improves MIT-skate when combined with M-LSTM.
  • Variant analysis: Max pooling performs better on MIT-skate, while average pooling performs better on Fis-V, providing no conclusive overall winner.The dataset-dependent reversal underscores the difficulty of the regression tasks.
  • Variant analysis: Linear SVR generally outperforms RBF SVR, but both perform worse than the proposed framework.
  • Variant analysis: C3D features produce better predictions than SENet features on Fis-V because clip-based representations better capture skater movement.C3D features are clip-based, whereas SENet features are static frame-based.
  • Target-score analysis: PCS correlations are generally better than TES correlations, while combining S-LSTM and M-LSTM improves PCS more consistently than TES.On TES, the combination does not significantly improve over either component alone, likely because the task focuses on technical movements and may receive redundant information.
  • Self-attention analysis: High-attention clips depict difficult technical movements, whereas low-attention clips depict movements treated as insignificant for TES.The qualitative examples use 16-frame clips and associate high attention with movements such as jumping on the same foot within a spin.

VI. CONCLUSION

The paper introduces the Fis-V dataset and two LSTM-based regression models for scoring figure-skating performances. Experiments on Fis-V and MIT-skate validate the effectiveness of the proposed methods.

  • Dataset: Fis-V is introduced as a dataset for figure-skating sports video analysis and scoring.
  • Models: The paper proposes Self-Attentive LSTM and Multi-scale Convolutional Skip LSTM models for the regression task.The two networks are also integrated into a single end-to-end framework.
  • Evaluation: Experiments on MIT-skate and Fis-V, including evaluations of variants, validate the effectiveness of the proposed methods.
Loading 1802.02774v3…