PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Content-Aware Video Analysis for Sports

Huang-Chia Shih

arXiv:1703.01170v1cs.CVcs.MM

TL;DR

Sports video analysis must help users access important information despite growing, diverse, and shared sports data, while prior surveys largely emphasized spatiotemporal methods and rarely addressed semantics. This paper surveys content-aware broadcast-sports analysis through content structure, hierarchical semantic levels, and future challenges. It organizes methods into object-, event-, and context-oriented groups and concludes by identifying prominent challenges and research directions.

  • Problem

    Sports data is increasingly large-scale, diversified, and shared, but rapidly accessing its most important information remains difficult; prior surveys rarely considered content-based semantics.

  • Method

    The paper comprehensively surveys broadcast-sports content analysis through fundamentals, the content pyramid, object-, event-, and context-oriented methods, and future challenges.

  • Results

    The survey reviews state-of-the-art sports content-analysis methods according to object-, event-, and context-oriented groups and categorizes prominent challenges and potential future directions.

  • Takeaways & Limitations

    Content-aware sports video analysis should connect semantic content organization with users’ intentions and varying information needs.

Abstract

from arXiv · show

Sports data analysis is becoming increasingly large-scale, diversified, and shared, but difficulty persists in rapidly accessing the most crucial information. Previous surveys have focused on the methodologies of sports video analysis from the spatiotemporal viewpoint instead of a content-based viewpoint, and few of these studies have considered semantics. This study develops a deeper interpretation of content-aware sports video analysis by examining the insight offered by research into the structure of content under different scenarios. On the basis of this insight, we provide an overview of the themes particularly relevant to the research on content-aware systems for broadcast sports. Specifically, we focus on the video content analysis techniques applied in sportscasts over the past decade from the perspectives of fundamentals and general review, a content hierarchical model, and trends and challenges. Content-aware analysis methods are discussed with respect to object-, event-, and context-oriented groups. In each group, the gap between sensation and content excitement must be bridged using proper strategies. In this regard, a content-aware approach is required to determine user demands. Finally, the paper summarizes the future trends and challenges for sports video analysis. We believe that our findings can advance the field of research on content-aware video analysis for broadcast sports.

I. INTRODUCTION

Sports video analysis has grown with expanding sports media and data, but rapidly finding important information remains difficult. This survey reframes the field around content structure, semantics, and content-aware systems for broadcast sports.

  • I. INTRODUCTION: Sports data analysis is becoming large-scale, diversified, and shared, increasing the need to access important information quickly.The paper situates this need within the growth of Internet video transmission, digital broadcasting, and sports media analytics.
  • A. Surveys and Taxonomies: Previous surveys emphasized spatiotemporal methodologies, whereas few considered content-based analysis or semantics.The paper adopts a different framework centered on the structure of content under different scenarios.
  • B. Scope and Organization of the Paper: The survey reviews broadcast-sports content analysis through fundamentals, a content hierarchical model, and challenges and future directions.Its content-pyramid review organizes semantic knowledge into object-, event-, and context-oriented groups.
  • I. INTRODUCTION: Content-aware retrieval should be designed according to users’ intentions.The paper connects content analysis with arranging and reasoning over video content for retrieval and related applications.
  • A. Content Pyramid: The content pyramid represents video information through video, object, action, and conclusion layers with decreasing information compactness from top to bottom.Conclusion frames summarize event tags and results, supporting game summaries based on event transcripts and outcomes.

B. Sports Genre Categorization

Sports genre classification is a common preliminary task for organizing sports media. Research covers field, posture, and racket sports, with most reviewed papers addressing baseball, basketball, soccer, and tennis.

  • B. Sports Genre Categorization: Broadcast sports are classified into three categories: field, posture, and racket.The classification is presented as a tree structure of sports genres and reviewed papers by publication year.
  • B. Sports Genre Categorization: More than 80% of reviewed papers addressed baseball, basketball, soccer, and tennis.This figure describes the distribution of reviewed work rather than the distribution of broadcast sports themselves.
  • B. Sports Genre Categorization: Sports genre classification methods have used text, audio, and visual features with classifiers including HMMs, naïve Bayesian classifiers, decision trees, and SVMs.Studies also considered schemes combining different classifiers.
  • B. Sports Genre Categorization: Combined NBC-HMM and NBC-SVM schemes were used for ball-sport genre classification and event detection.The NBC-SVM study categorized tennis, basketball, volleyball, soccer, and table tennis.
  • B. Sports Genre Categorization: Multimodal auxiliary data from sensors, visual information, and audio can support sports-type classification in mobile-video applications.Other work classified unedited home videos into five genres using salient low-level features and ensemble methods.

C. Overview of Sports Video Analysis

Content-aware sports video analysis organizes processing from feature extraction through information reasoning and knowledge arrangement, using semantic levels and object-, event-, and context-oriented techniques. Object analysis supports recognition and tactic analysis, but occlusion, camera motion, blur, and group movement constrain extraction and tracking.

  • Overview: A content-aware video analysis system generally performs feature extraction, information reasoning, and knowledge arrangement across levels of the content pyramid.Information reasoning addresses low-, mid-, and high-level video data.
  • Overview: The survey categorizes techniques by semantic level into highlight detection and event recognition, object detection and action recognition, and contextual inference and semantic analysis.
  • Object-oriented analysis: Object-oriented analysis uses intraobject posture and interobject event relationships to support action recognition across diverse sports.Single-object applications require accurate posture and movement, while two-object applications analyze relationships between objects.
  • Challenges: Occlusion, camera motion, blur, sudden movement, and group-based continual occlusion remain practical constraints on object detection and tracking.Reported responses include multiple cameras, 3D information, region-based detection, and occlusion reasoning.
  • Object-oriented analysis: Object extraction and tracking support event reasoning and tactic analysis by modeling players, balls, trajectories, and interactions in sports video.Trajectory data can identify tactics, while attention scores can estimate frame and clip excitement.

3) Naming Objects

Naming objects in sports video remains difficult because athlete faces are not always observable, motivating text-based identification methods. A CNN-based clothing-number recognizer achieved an 83% recognition rate under a specified augmentation and dropout setting.

  • 3) Naming Objects: Athlete faces are not continuously observable because players move unpredictably through three-dimensional space.
  • 3) Naming Objects: Video OCR and scene text recognition provide alternative name-assignment cues by reading text or numbers printed on athletes’ clothes.
  • 3) Naming Objects: 83% recognition rate was achieved by a CNN clothing-number recognizer using data augmentation and no dropout.The compared feature-vector models treated each number as one class or each digit as a class.

4) Action Recognition

Sports action-recognition research spans posture, movement, human–object context, sensing, feature representation, classification, and benchmark datasets. The survey emphasizes seven sports action datasets and reviews methods across multiple sports and application settings.

  • a) Posture and movement: Action recognition in sports is organized around individuals, interactions between objects, and interactions between people.
  • b) Feature representation: Deep-learning approaches learn discriminative features, sometimes combining handcrafted and learned descriptors for action recognition.
  • a) Posture and movement: Mutual context models use object–pose co-occurrence and spatial relationships so objects and human poses can facilitate each other’s recognition.
  • a) Posture and movement: Sensor-based systems can track baseball swings from multidimensional physiological data and cluster basic movement patterns to detect mistakes and provide feedback.
  • a) Posture and movement: Domain knowledge and human behavior analysis can improve robustness when partial tracking errors or missing information destabilize recognition.
  • d) Data sets: The survey reports seven sports action datasets, including two with image sources and action markers and five consisting of video clips.
  • d) Data sets: Sports-1M contains greater than 1.1 million annotated YouTube videos across 487 sports, with approximately 74.4% recognition from a two-stream CNN and temporal feature pooling.

B. Highlight Detection and Event Recognition

Large-scale content-based multimedia mining has expanded, but systematic surveys specifically focused on sports videos remain limited. The section therefore reviews event-oriented studies by data source, methodology, sports genre, desired event, and application field.

  • The survey reviews sports event-analysis studies across data sources, methodologies, sports genres, desired events, and application fields.

1) Scene Change Detection

Scene-change detection divides long sports videos into shots that support event analysis. Shots are treated as segments containing identical semantic concepts, enabling subsequent event classification.

  • Shot-boundary detection partitions long video sequences into camera shots, and each shot contains identical semantic concepts for event classification.

2) Play-and-break Analysis

Generic analysis across all sports is difficult because events are domain-dependent. Play-and-break analysis offers a compromise by modeling games through play and nonplay segments that support summarization.

  • A generic model for all sports is difficult because the meaning of an event depends on the domain.
  • Play-and-break analysis models sports games using plays and breaks, with detected play information supporting game summarization.

3) Keyframe Determination

Keyframe determination represents each video segment after shot boundaries are retrieved. The reviewed approach detects changes in superimposed captions and uses rule-based decisions to select meaningful event frames.

  • Keyframe detection represents the status of each video segment after shot-boundary retrieval.
  • Liang et al. detect superimposed-caption data changes and apply rule-based decisions to identify meaningful baseball events.Their method considers outs, scores, and base occupation, allowing simple representative-frame selection.

4) Highlight Detection

Highlight detection extracts critical sports events by combining audio and visual information with specialized features and analyzers. Reviewed visual features span captions, playfields, camera motion, ball and player positions, and replays, with some systems also ranking extracted highlights.

  • Highlight extraction targets critical sports events using specific features and analyzers across audio and visual modalities.
  • Visual features for highlight detection comprise captions and text, playfields, camera motion, ball positions, player positions, and replays.
  • Caption detection and scoreboard recognition can identify event frames from overlaid text and game-state information.
  • Player, referee, and ball position time series can support prediction of key activities and dominance conditions in soccer.
  • Support vector regression can construct nonlinear highlight-ranking models for racket-sports videos.

5) Use of Domain Knowledge

Domain knowledge and temporal shot patterns support sport-specific event recognition, but unusual interruptions and uncertain playfield conditions complicate semantic analysis. Context from captions and visual cues further supplies game-state meaning.

  • 5) Use of Domain Knowledge: Shot-boundary segmentation and domain knowledge help recognize events such as goals, cards, and penalties through temporal action modeling.
  • 5) Use of Domain Knowledge: Sports video analysis is domain-specific because temporal patterns vary by sport.Systems may exploit identical game structures, while domain knowledge can improve recognition accuracy.
  • 5) Use of Domain Knowledge: Soccer goals follow shot patterns involving gate appearance, close-up views, scorer celebration, scene transitions, and replays.
  • 5) Use of Domain Knowledge: Baseball home-run recognition can combine pre-pitch layouts, announcer audio, rapid camera motion, and audience or fielder views.
  • 5) Use of Domain Knowledge: Basketball event classification uses scoreboard changes, close-ups, player trajectories, and ball-passing paths because regular temporal patterns are fewer.
  • 5) Use of Domain Knowledge: Context-based semantic analysis can assess unusual events despite uncertainty from birds, wind, transcoding, streakers, arguments, and fights.
  • 5) Use of Domain Knowledge: Context comprises event outcomes and match summaries obtained from captions or visual cues.
  • 5) Use of Domain Knowledge: Superimposed caption boxes convey ongoing game status, motivating methods for caption detection, reading, and semantic interpretation.

2) Text Analytics

Text analytics in sports video analysis uses detection, recognition, extraction, and semantic-event processing to connect textual information with game events. The paper identifies media-cloud scalability, deeper context discovery, and robust human-action recognition as future directions.

  • 2) Text Analytics: Multiframe averaging and binarization can support video-text detection, localization, extraction, and recognition.
  • 2) Text Analytics: A naïve Bayesian classifier can identify an optimal match event after detecting the game start time in a crawled match report.
  • 2) Text Analytics: Semantic event extraction can parse events hierarchically and combine text analysis, video analysis, and alignment for moment and boundary detection.
  • 2) Text Analytics: Future directions are grouped into media-cloud analysis, large-scale learning for deeper context discovery, and robust visual features for human action recognition.
  • 2) Text Analytics: Crowdsourced mobile sports recordings remain difficult to combine because sports videos lack a continuous, smooth audio stream for stitching.
  • 2) Text Analytics: Content-aware scalability must address users’ desired competition status through fine-grained transcoding across bandwidth conditions and end devices.

B. Large-scale Machine Learning Techniques for Deeper

Large-scale machine learning has advanced sports video understanding, but domain-specific semantics and structures still make unified, transferable analysis difficult. The survey organizes content-aware methods and identifies future challenges for diverse sports.

  • Big data analytics and deep learning support discovering latent semantic concepts and recognizing actions in sports video.Deep learning methods, including CNNs, restricted Boltzmann machines, autoencoders, and sparse coding, have been applied to visual understanding and action recognition.
  • Sports videos require domain-specific semantic concepts, structures, and features, leaving the semantic gap unresolved.The challenge is to autoencode and transform domain-specific features and models across sports domains.
  • Domain adaptation remains challenging because sports training and testing data may differ across seasons, sports, and tactical contexts.The survey identifies knowledge interpolation across domains as a promising direction, including transfer between seasons and related sports.
  • Object trajectories and CNN feature hierarchies improve the detail, accuracy, and robustness of action recognition and tracking.Reviewed approaches include dense trajectories, improved trajectories, spatiotemporal convolution kernels, and layer-wise CNN correlation filters.
  • The survey reviews content-aware sports analysis through fundamentals, a content hierarchical model, object-, event-, and context-oriented methods, and future challenges.It frames the review as a comprehensive survey intended to advance content-aware video analysis for sports.
Loading 1703.01170v1…