PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Summarization Using Deep Neural Networks: A Survey

Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, Ioannis Patras

arXiv:2101.06072v2cs.CVcs.LGcs.MM

TL;DR

Video summarization must condense increasingly abundant video into informative synopses, while annotation is costly and summaries are subjective. This survey taxonomizes and systematically reviews deep-learning methods, evaluates their performance and protocols, and identifies future directions. It highlights attention and memory mechanisms among strong supervised methods and GAN-based approaches as promising for unsupervised summarization.

  • Problem

    Video summarization needs to select informative content from large video collections, but ground-truth summaries are costly to produce and can vary substantially across annotators.

  • Method

    The paper formulates the task, describes typical pipelines, proposes a taxonomy, systematically reviews deep-learning methods, and examines evaluation protocols and performance.

  • Results

    The survey identifies tailored attention or memory networks among the best-performing supervised approaches and GANs as a promising direction for unsupervised summarization.

  • Takeaways & Limitations

    Future work should further investigate unsupervised and semi/weakly supervised learning because annotation is expensive and human summaries are subjective and variable.

Abstract

from arXiv · show

Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content. Several approaches have been developed over the last couple of decades and the current state of the art is represented by methods that rely on modern deep neural network architectures. This work focuses on the recent advances in the area and provides a comprehensive survey of the existing deep-learning-based methods for generic video summarization. After presenting the motivation behind the development of technologies for video summarization, we formulate the video summarization task and discuss the main characteristics of a typical deep-learning-based analysis pipeline. Then, we suggest a taxonomy of the existing algorithms and provide a systematic review of the relevant literature that shows the evolution of the deep-learning-based video summarization technologies and leads to suggestions for future developments. We then report on protocols for the objective evaluation of video summarization algorithms and we compare the performance of several deep-learning-based approaches. Based on the outcomes of these comparisons, as well as some documented considerations about the amount of annotated data and the suitability of evaluation protocols, we indicate potential future research directions.

I. INTRODUCTION

Automatic video summarization addresses the challenge of navigating rapidly expanding video collections by producing concise synopses of important content. Deep-learning-based methods typically extract visual features, learn frame importance, and generate storyboards or skims, with approaches differing by training strategy and modality.

  • Motivation: Automatic video summarization generates concise synopses that help viewers browse and navigate large video collections.Its applications include media-asset indexing, browsing, retrieval, promotion, and improved viewing experiences.
  • Related work: Earlier surveys classified video summarization by scenario, visual content, utility, structure, redundancy, detected objects or events, and multimodal integration.The surveyed literature argues that prior surveys did not comprehensively present current developments in generic deep-learning-based summarization.
  • Summary forms: A video storyboard contains representative key-frames, whereas a video skim concatenates key-fragments chronologically into a shorter video.Both summary types select informative and important parts of the original video.
  • Analysis pipeline: The typical deep-learning pipeline represents video content with frame-level feature vectors, feeds them to a trained deep summarizer, and outputs selected frames or fragments.Feature extraction may process every frame or use sampling such as 2 frames per second.
  • Analysis pipeline: Video skims additionally require coherent, non-overlapping temporal segmentation and fragment-level importance scores derived from frame-level scores.The selected fragments are concatenated in chronological order for seamless presentation.
  • Taxonomy: Deep-learning-based methods are categorized by training strategy into supervised and unsupervised approaches, and by modality into unimodal and multimodal approaches.Supervised methods use human annotations, while unsupervised methods train on collections of original videos without ground-truth summaries.

III. DEEP LEARNING APPROACHES

The survey organizes deep-learning-based video summarization research into supervised, unsupervised, weakly supervised, and multimodal approaches. It uses these classes to structure a systematic review of the relevant literature.

  • Method taxonomy: The review covers supervised, unsupervised, and weakly supervised methods that rely solely on visual content.It then reports on multimodal approaches using additional information beyond visual analysis.

A. Deep Learning Basics

Deep learning uses multiple nonlinear network layers to learn representations from data, and video summarization is one application of several deep-learning architectures. The field also faces challenges involving data, optimization, compactness, stability, and interpretability.

  • Deep learning definition: Deep learning uses multiple network layers to perform nonlinear processing and learn multiple levels of data representation.The survey distinguishes supervised, semi-supervised, and unsupervised learning.
  • Architecture classes: Architectures used in deep learning include convolutional, recurrent, generative adversarial, graph-convolutional, autoencoder, and probabilistic networks.The survey lists these among several broad architecture classes.
  • Challenges: Deep-network research faces challenges in learning from sparse or noisy data, optimization, compact deployment, stability analysis, and explainability.These challenges are described as critical to training deep networks.
  • Video summarization taxonomy: The survey taxonomy organizes existing deep-learning-based video summarization algorithms into a structured tree of learning approaches and implementation techniques.The taxonomy provides the basis for the subsequent literature review.

B. Supervised Video Summarization

Supervised video summarization learns frame or fragment importance from human-generated ground truth, modeling temporal or spatiotemporal dependencies. Related supervised pipelines also use adversarial learning to make generated summaries resemble human summaries.

  • Temporal dependency: Temporal supervised methods learn frame importance by modeling dependencies among video frames against ground-truth user-preference scores.Early approaches cast summarization as a structured prediction problem.
  • Spatiotemporal dependency: Spatiotemporal supervised methods extend temporal modeling by representing spatial relationships among objects as well as dependencies across frames.Their inputs include video frames and ground-truth frame-importance data.
  • Spatiotemporal dependency: Convolutional LSTMs can model spatiotemporal relationships while next-frame prediction and shot detection mechanisms enhance summary visual diversity.An encoder-decoder architecture combines these components with frame-importance estimation.
  • Attention-based selection: Self-attention can use human-generated ground truth to estimate intra-shot importance and select key-frames or key-fragments.A motion curve and shot-segmentation stage precede this selection process.
  • Adversarial learning: Adversarial supervised methods use a summarizer as a generator and a discriminator that distinguishes machine-generated summaries from human-generated summaries.The discriminator compares predicted frame-level importance scores with optimal user-preference scores; some methods use its output as a reinforcement-learning reward.

C. Unsupervised Video Summarization

Unsupervised video summarization methods learn without ground-truth summaries by reconstructing the original video, optimizing hand-crafted summary properties, or modeling important objects and motions.

  • Reconstruction-based approaches: GAN-based methods learn summaries by making summary-based reconstructions difficult to distinguish from the original video.LSTM-based selectors estimate frame importance, while autoencoders, attention mechanisms, Actor-Critic models, or self-attention assist reconstruction or selection.
  • Reconstruction-based approaches: Adversarial methods seek summaries that preserve enough information to reconstruct the overall video content.Representative approaches combine LSTM selectors with VAEs, discriminators, attention autoencoders, or cycle-consistent adversarial objectives.
  • Property-driven approaches: Reinforcement-learning methods use hand-crafted rewards to target desired summary properties instead of relying primarily on reconstruction-based GAN objectives.Rewards can quantify representativeness, diversity, uniformity, spatiotemporal pattern retention, or summary-based reconstruction.
  • Object-oriented approaches: Object-oriented summarization preserves fine-grained semantic and motion information by modeling important visual objects and their key-motions.The method preprocesses videos to identify important objects and key-motions, then represents the video with object motion clips.

D. Weakly-supervised Video Summarization

Weakly supervised methods reduce reliance on extensive human annotations by learning from imperfect metadata, sparse labels, related videos, or annotations transferred across domains.

  • Definition and motivation: Weakly supervised summarization uses less-expensive labels while assuming they are imperfect substitutes for full human annotations.Examples include video-level metadata, sparse frame annotations, and category labels used to train summarization models.
  • Semantic and category-driven methods: Video metadata can define video categories that guide category-driven summarization using semantically related content.Metadata such as a title can categorize videos, while summaries from similar videos help identify category-relevant content.
  • Transfer and sparse supervision: Other weakly supervised approaches transfer knowledge from annotated related domains or use weakly and sparsely labeled data with reinforcement learning.Cross-domain feature embedding and transfer learning support knowledge transfer, while tailored rewards guide learning from limited annotations.
  • Multimodal approaches: Multimodal methods combine video frames with ground-truth importance data and textual metadata to predict frame importance and generate summaries.Inputs may include frame sequences, user-provided importance scores, titles, and descriptions; generated summaries are then compared with ground truth.
  • Research direction: The survey recommends further exploration of fully unsupervised and semi- or weakly supervised methods because generating ground-truth summaries is expensive and laborious.The recommendation is presented despite supervised algorithms remaining the major research direction.

IV. EVALUATING VIDEO SUMMARIZATION

Video summarization evaluation relies on datasets with ground-truth summaries or frame- and fragment-level importance annotations, with four datasets prevailing in the literature.

  • Common datasets: Four datasets prevail: SumMe, TVSum, OVP, and Youtube.These datasets provide the principal annotated resources used in the video summarization bibliography.
  • Common datasets: SumMe contains 25 diverse 1-to-6-minute videos annotated by 15 to 18 users with key-fragments and summaries lasting 5% to 15% of each video.Its videos include both first-person and third-person views, and each video has multiple fragment-level user summaries.
  • Common datasets: OVP and Youtube each contain 50 videos annotated by five users with key-frame sets.OVP videos last 1 to 4 minutes, while Youtube videos last 1 to 10 minutes and cover diverse content categories.
  • Additional datasets: Less commonly used datasets include CoSum, MED-summaries, VTW, LoL, and FVPSum.CoSum supports video co-summarization with 51 approximately four-minute videos annotated by three users using key-fragments.
  • Dataset selection: The survey focuses on datasets suitable for evaluation because they contain ground-truth summaries or frame- or fragment-level importance annotations.Other datasets may be used for network pre-training but are outside the survey’s evaluation focus.

B. Evaluation protocols and measures

Evaluation protocols evolved from time-consuming human judgments of static key-frame summaries toward objective measures and preference-alignment methods for dynamic video skims.

  • Static summaries: Early static summaries used representative key-frames and were evaluated by human judges for relevance, redundancy, informativeness, or enjoyability.User studies can assess summary quality but are time-consuming and difficult to reproduce or reuse for comparisons.
  • Static summaries: Objective evaluation replaced some user studies by comparing key-frame summaries with ground-truth summaries using measures such as Fidelity and Shot Reconstruction Degree.The Comparison of User Summaries protocol provided another ground-truth-based evaluation approach.
  • Dynamic summaries: Recent algorithms generate dynamic video skims by selecting representative fragments and joining them into shorter videos.Their evaluation assesses how well the skims align with human preferences, extending beyond earlier qualitative storyboard evaluation.
  • Dynamic summaries: Evaluation methodologies for video skims focus on alignment with human preferences rather than only qualitative judgments.This shift reflects the transition from key-frame storyboards to shortened sequences of video fragments.

2) Evaluating video skims:

Video-skims are evaluated by matching selected key-fragments against user summaries and computing agreement, with protocols differing in how user annotations and video fragmentation are handled.

  • Key-fragment matching: An optimal subset of consecutive, non-overlapping video fragments is selected as the skim and compared with user-generated summaries using pairwise F-Score.Fragmentation enables matching automatically created and user-generated key-fragment summaries.
  • Key-fragment matching: Precision and Recall are computed per frame against each available user summary before calculating the video-level F-Score.The methodology preserves multiple user summaries for each test video.
  • Benchmark protocols: The SumMe and TVSum evaluation datasets have been jointly used with revised video segmentations and conversions from frame-level scores to key-fragment summaries.These conversions were documented in supplementary material for one evaluated method.
  • Alternative protocols: Single-ground-truth evaluation averages users’ frame-level annotations, but it does not preserve each user’s original opinion and therefore provides less firm evaluation.This approach is used by only a few methods.
  • Alternative protocols: Rank-correlation evaluation avoids predefined video fragmentation but is restricted to TVSum-style annotations and methods producing frame-level importance scores.The reported coefficients are Kendall τ and Spearman ρ.
  • Alternative protocols: “Performance over Random” evaluates five public methods across 50 randomly created SumMe and TVSum data splits while accounting for split difficulty.The protocol uses F-Score and incorporates split-difficulty information during evaluation.

A. Quantitative comparisons

Across SumMe and TVSum, the survey compares deep-learning video summarization methods under multiple evaluation protocols and identifies performance patterns alongside substantial protocol inconsistencies.

  • Supervised approaches: The strongest supervised methods use tailored attention or memory networks to capture variable- and long-range temporal dependencies.Examples include VASNet, H-MAN, SUM-GDA, DASP, CSNetsup, and SMN.
  • Cross-dataset behavior: Some methods perform strongly on one dataset but very poorly or randomly on the other, indicating possible dataset-specific adaptation.The survey cites MAVS, DASP, M-AVS, and TTH-RNN as examples.
  • Unsupervised approaches: GAN-based methods dominate the strongest fully unsupervised results, including AC-SUM-GAN, CSNet, SUM-GAN-AAE, and SUM-GAN-sl.The comparison covers unsupervised methods evaluated on the prevalent SumMe and TVSum datasets.
  • Architectural patterns: Attention mechanisms improve reported performance, with SUM-GAN-AAE outperforming SUM-GAN-sl on SumMe while matching it on TVSum.The change replaces a Variational Auto-Encoder with a deterministic Attention Auto-Encoder.
  • Learning strategies: Reward-function and reinforcement-learning methods are less competitive than GAN-based methods, especially on SumMe.The comparison specifically identifies DR-DSN and EDSN as less competitive.
  • Evaluation limitations: Performance comparisons are limited because studies use inconsistent random splits, sometimes with overlap, and often omit split information.This diversity prevents perfectly accurate comparisons between algorithms.

B. Qualitative Comparisons and Demos

The survey supplements numerical comparisons with qualitative summaries of a SumMe video, showing how five methods select different shots and frames. It also points readers to an online service that generates platform-tailored summaries.

  • SUM-GAN-AAE and VASNet select the highest-importance shots and achieve the highest F-score for SumMe video #19.Their selected key-frames are diverse and provide a good overview of the plane’s landing.
  • DR-DSN selects nearly the same shots as SUM-GAN-AAE and VASNet but achieves slightly lower performance.
  • SUM-GAN-sl focuses less on the video’s main event, while dppLSTM selects many frames that miss the video’s central point.
  • An online video summarization service based on SUM-GAN-AAE can create multiple summaries tailored to different sharing and social-networking channels.

VI. FUTURE DIRECTIONS

The survey identifies future directions spanning annotation-efficient learning, richer summary constraints, improved GAN training, multimodal modeling, and more reliable evaluation protocols.

  • VI. FUTURE DIRECTIONS: Future methods should reduce reliance on large human-annotated datasets through fully unsupervised, semi-supervised, or weakly supervised learning.This could reduce laborious annotation and support adaptation to different domains and application scenarios.
  • VI. FUTURE DIRECTIONS: Unsupervised systems could combine reconstruction with mechanisms enforcing additional summary criteria, including visual diversity.
  • VI. FUTURE DIRECTIONS: Semi- and weakly supervised systems could align summaries with user-specified rules through textual queries or online interaction.
  • VI. FUTURE DIRECTIONS: Improving GAN-based summarization training requires addressing convergence conditions and mode collapse, potentially with augmented data and curriculum learning.
  • VI. FUTURE DIRECTIONS: Future architectures could overcome RNN training and memory limitations while combining visual and audio representations to select suitable summary segments.
  • VI. FUTURE DIRECTIONS: Better evaluation protocols are needed because existing imperfections limit accurate comparisons between summarization methods.

VII. CONCLUSIONS

The survey systematically reviews deep-learning-based video summarization and concludes that attention-enhanced recurrent supervised methods and GAN-based unsupervised methods are especially promising. It also highlights annotation scarcity, inconsistent evaluation protocols, and the need for practical integration as continuing concerns.

  • VII. CONCLUSIONS: The survey provides a systematic review of the deep-learning-based video summarization landscape, its evolution, open issues, and future directions.
  • VII. CONCLUSIONS: Best-performing supervised methods model variable-range temporal dependencies with recurrent neural networks and tailored attention mechanisms.
  • VII. CONCLUSIONS: GAN-based unsupervised methods combine adversarial learning with attention or Actor-Critic models and achieve performance comparable to state-of-the-art supervised approaches.
  • VII. CONCLUSIONS: Diverse data-splitting and evaluation practices, including differently difficult random splits, raise concerns about the accuracy of cross-paper performance comparisons.
  • VII. CONCLUSIONS: The survey recommends practical integration of summarization technologies into tools supporting time-efficient video adaptation and reuse by media organizations.
Loading 2101.06072v2…