PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Summarization with Long Short-term Memory

Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman

arXiv:1605.08110v2cs.CVcs.LG

TL;DR

Video summarization requires selecting keyframes or key subshots while accounting for dependencies across sequential frames and limited annotated data. The paper uses LSTM-based structured prediction, enhanced with DPP and domain adaptation across heterogeneous datasets, and reports state-of-the-art or best benchmark performance, though qualitative failures remain in some diverse videos.

  • Problem

    Video summarization must make interdependent sequential selections while obtaining enough annotated data to train supervised models.

  • Method

    The paper develops LSTM-based supervised models for keyframe, keyshot, and importance annotations, combines LSTMs with DPP, and uses domain adaptation across datasets.

  • Results

    The models achieve state-of-the-art or best results on two benchmark datasets, with accuracy improving when heterogeneous annotated datasets are combined using transfer learning or augmentation.

  • Takeaways & Limitations

    Sequential modeling captures variable-range interdependencies, while DPP adds inter-frame repulsiveness and auxiliary datasets help mediate LSTMs’ annotation demands.

  • Takeaways & Limitations

    A qualitative failure case shows dppLSTM can miss temporally clustered important beginning frames when enforcing diversity, reducing recall.

Abstract

from arXiv · show

We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots. Casting the problem as a structured prediction problem on sequential data, our main idea is to use Long Short-Term Memory (LSTM), a special type of recurrent neural networks to model the variable-range dependencies entailed in the task of video summarization. Our learning models attain the state-of-the-art results on two benchmark video datasets. Detailed analysis justifies the design of the models. In particular, we show that it is crucial to take into consideration the sequential structures in videos and model them. Besides advances in modeling techniques, we introduce techniques to address the need of a large number of annotated data for training complex learning models. There, our main idea is to exploit the existence of auxiliary annotated video datasets, albeit heterogeneous in visual styles and contents. Specifically, we show domain adaptation techniques can improve summarization by reducing the discrepancies in statistical properties across those datasets.

1 Introduction

Video summarization is framed as supervised structured prediction over sequential frames, where models must capture complex dependencies and annotated data is scarce. The paper proposes LSTM-based models, combines them with DPP for diversity, and uses heterogeneous datasets with domain adaptation to improve training.

  • Motivation: Video summarization selects keyframes or contiguous key subshots to compactly depict important events for browsing large video collections.The paper focuses on these two common forms of summarization.
  • Problem formulation: The task outputs frame-selection decisions whose dependencies span the video, so selections cannot be made locally or independently.Human viewers use semantic understanding and unfolding storylines when deciding which frames to retain.
  • Proposed approach: LSTMs model variable-range dependencies by using memory cells to learn when to forget or incorporate past events during summarization.This design targets intertwined short-range and long-range relationships in video sequences.
  • Contributions: The paper introduces vsLSTM and combines LSTM modeling with DPP to encourage diverse selected frames.The sequential modeling component is reported as essential relative to MLPs using neighboring frames.
  • Contributions: The resulting models achieve the best results on two challenging benchmark datasets and benefit from combining heterogeneous annotated video datasets with domain adaptation.Domain adaptation reduces discrepancies in statistical characteristics across datasets.

2 Related Work

Prior video summarization methods use unsupervised criteria or supervised human-created summaries, while this paper addresses the gap of learning both short- and long-range sequential dependencies with LSTMs.

  • Existing approaches: Video summarization methods are broadly divided into unsupervised approaches using manually designed criteria and supervised approaches learning from human-edited summaries or importance ratings.Supervised methods are described as promising relative to traditional unsupervised methods.
  • Existing approaches: Common selection criteria include relevance, representativeness or importance, and diversity or coverage.Some methods also use auxiliary web images or video categories.
  • Supervised methods: Supervised methods can align summaries with human judgments by learning from ground-truth summaries and can use DPP to model representative, diverse subsets.Prior supervised approaches either combine hand-crafted criteria or apply DPP.
  • Research gap: Earlier video-summarization work did not use LSTMs to learn both short-range and long-range dependencies in sequential video frames.Sequential DPPs instead rely on predefined temporal structures, whereas LSTMs use data-dependent switching.
  • Research gap: The paper presents LSTMs as a first application to video summarization, building on their use for video highlights, visual attention, and natural-language video description.Its motivation is the flexibility of LSTMs in capturing sequential structure.

3 Approach

The approach treats video summarization as learning frame-level outputs from sequential visual features, using LSTM models to capture dependencies across time and DPPs to promote diverse selections.

  • 3.1 Problem Statement: Video summarization maps frame sequences to keyframes, keyshots, or frame-level importance scores used as learning targets.The models support binary keyframe labels, binary subshot labels, and continuous frame-level importance annotations.
  • 3.2 Long Short-Term Memory (LSTM): LSTMs use memory cells and input, forget, and output gates to regulate information from current inputs and prior states.The forget gate provides a data-dependent mechanism for selectively incorporating past and present information.
  • 3.3 vsLSTM: vsLSTM uses bidirectional LSTM layers to model dependencies in both past and future directions, then combines their hidden states and visual features with an MLP.The MLP produces a scalar representing the likelihood that each frame should be included or its importance to the summary.
  • 3.4 Enhancing vsLSTM by Modeling Pairwise Repulsiveness: dppLSTM combines vsLSTM with a determinantal point process to model long-range dependencies and pairwise frame-level diversity.DPP probabilities favor subsets with significant pairwise dissimilarity, complementing vsLSTM’s frame-level importance predictions.
  • 3.4 Enhancing vsLSTM by Modeling Pairwise Repulsiveness: The dppLSTM kernel separates frame importance from frame similarity using two MLPs, and this decomposition performs better empirically than a single shared MLP.The two MLPs are fI(·) for frame-level importance and fS(·) for similarity.
  • Learning: dppLSTM is trained stage-wise, first fitting vsLSTM components and then maximizing the DPP likelihood of target keyframe subsets.Its selected-frame outputs are converted into keyshot-based summaries for evaluation.

4 Experimental Results

Experiments evaluate LSTM-based video summarization across datasets, supervision settings, feature types, annotation formats, domain adaptation, and qualitative cases. Results show benefits from sequential modeling and more heterogeneous annotations, while performance depends on video characteristics and the balance between importance and diversity.

  • Experimental setting: The study evaluates models on SumMe and TVSum, using keyshot-based precision, recall, and F-score under a 15% duration constraint.It also incorporates auxiliary annotated datasets and multiple supervision settings, including canonical, augmented, and transfer learning.
  • Main results: On TVSum, both proposed methods outperform the state-of-the-art, whereas on SumMe they underperform, likely because fewer annotated samples are available.The comparison is reported for the canonical supervised-learning setting.
  • Main results: Increasing annotated data improves performance, with augmented training on all four datasets producing the best results and transfer learning improving over canonical training despite dataset differences.The four datasets are heterogeneous in visual styles and contents.
  • Sequence modeling: With more annotations, vsLSTM outperforms MLP baselines, supporting the importance of modeling sequential video structure while recognizing that LSTMs require substantial supervision.In the canonical setting, MLP-based approaches outperform vsLSTM.
  • Representation and supervision: Deep features modestly improve performance over shallow features, while frame-level importance scores consistently outperform alternative annotation formats.dppLSTM with shallow features still outperforms the prior TVSum result using the same shallow features.
  • Domain adaptation: Domain adaptation improves summary F-score after aligning dataset covariances, with especially pronounced gains on the smaller SumMe dataset.The adapted datasets improve performance in both transfer and augmented settings.
  • Qualitative results: Qualitative examples show dppLSTM capturing important temporal context and selecting similar but important distant subshots, though it can lose recall when many important frames are crowded together.The authors expect better performance when video contents change smoothly; rapidly changing, diverse videos may require higher-level semantic cues.

5 Conclusion

The paper develops LSTM-based supervised approaches for automatic video summarization, combining variable-range dependency modeling with DPP-based diversity and heterogeneous auxiliary datasets. These models outperform competing methods on two challenging benchmarks and motivate richer sequential models for future work.

  • LSTM-based models outperform competing methods on two challenging video-summarization benchmarks.The conclusion attributes this performance to LSTMs’ capacity to capture variable-range inter-dependencies and DPP’s explicit modeling of inter-frame repulsiveness.
  • DPP complements LSTMs by explicitly modeling inter-frame repulsiveness to encourage diverse selected frames.
  • Heterogeneous annotated video datasets can be exploited to mediate the large-sample demand of LSTM training.The paper presents this as a promising direction for combining a broader collection of video datasets.
  • Future work could develop sequential models that encode semantic understanding of video contents to guide summarization and other visual-analytics tasks.

Supplementary Material: Video Summarization with Long Short-term Memory

The supplementary material documents annotation formats, procedures for converting among them, dataset details, model-training requirements, and implementation discussions. Its conversion framework supports combining datasets and training summarization algorithms across heterogeneous ground-truth formats.

  • The supplement covers ground-truth conversion, dataset details, LSTM-model objectives, network comparisons, implementation, and additional video-summarization discussions.
  • Video-summarization datasets commonly provide keyframes, interval-based keyshots, or frame-level importance scores.
  • The conversion procedure enables multiple datasets to enlarge training sets or supports supervised algorithms requiring different ground-truth formats.Conversions are applied to training ground truths, while testing generally uses user-generated summaries in their original formats.
  • Tables 1 and 2 illustrate annotation formats and the conversion procedure using six-frame sequences and temporally segmented intervals.
  • Keyframes are converted into keyshots by temporal segmentation, assigning score 1 to intervals containing keyframes and 0 otherwise.
  • Candidate keyshot intervals are ranked by keyframe density and selected under a duration threshold.The procedure may use a knapsack algorithm to keep total keyshot duration below the threshold.
  • Keyshots can produce keyframes or frame-level scores, while frame-level scores can be averaged within intervals to select keyshots.

B Details of the datasets

The experiments involve four video-summarization datasets: SumMe, TVSum, OVP, and Youtube. OVP and Youtube are used only to augment the training set.

  • The experiments use SumMe, TVSum, OVP, and Youtube, with OVP and Youtube reserved for training-set augmentation.

B.1 Training ground truths

Training requires one ground truth per video, with direct annotations available for SumMe and TVSum but not for OVP and Youtube. The latter datasets therefore require constructed single ground-truth keyframe sets, and training formats are converted as needed.

  • Training requires a single ground truth for each video; SumMe and TVSum provide one directly, whereas OVP and Youtube do not.
  • For OVP and Youtube, a single ground-truth keyframe set is created from multiple user annotations following prior work.
  • The proposed models and baselines require different training-ground-truth formats, so formats are converted when dataset annotations do not match.KTS is used for temporal segmentation across all datasets.

B.2 Testing ground truths for TVSum

TVSum evaluation converts user-annotated frame-level importance scores into keyshot-based summaries, while the proposed models are trained using structured subset targets and DPP likelihood optimization.

  • TVSum supplies multiple frame-level importance-score sequences annotated by different users for each video.
  • The annotations are converted into keyshot-based summaries for evaluation using a threshold equal to 15% of the original video length.
  • The proposed LSTM-based models require training ground truths in formats specified for vsLSTM and dppLSTM.
  • The DPP stage defines a ground set of video frames, a target keyframe subset, and a kernel matrix whose likelihood is optimized over model parameters.

C.2 Generating shot-based summaries for vsLSTM and dppLSTM

The models produce frame-level outputs but are evaluated as keyshot summaries by converting selected frames or scores under a fixed duration threshold and, for dppLSTM, supplementing short outputs.

  • vsLSTM predicts frame-level importance scores, whereas dppLSTM selects keyframes using approximate MAP inference before keyshot conversion.
  • vsLSTM keyshots are converted with a total-duration threshold of 15% of the original video length for both datasets.
  • dppLSTM inference often produces high-precision, low-recall summaries averaging 10% duration, so additional keyshots are added using the importance output of fI(·).

D Comparing different network structures for dppLSTM

The comparison isolates two dppLSTM network structures and finds that the full model, which separately models importance and similarity, outperforms the single-MLP variant.

  • dppLSTM uses two MLPs: fI(·) outputs frame-level importance scores, while fS(·) outputs similarity features.
  • dppLSTM-single learns only fS(·) before stacking with a DPP and uses L’s diagonal values to supplement insufficient-duration summaries.
  • dppLSTM obviously outperforms dppLSTM-single across the settings compared in Table 5.
  • The authors surmise that separately modeling quality and diversity benefits DPP summarization, particularly when each factor can be pretrained.

E Other implementation details

Implementation uses consistent sampling, network dimensions, losses, and validation stopping criteria, with temporal structure treated as central to video summarization.

  • The implementation details cover the proposed models vsLSTM and dppLSTM alongside the MLP-Frame and MLP-Shot baselines.
  • Videos are uniformly subsampled to 2 fps for frame-feature models, while MLP-Shot segments videos into shots averaging about 5 seconds.
  • All models use 256 hidden units or output dimensions for the specified MLP and LSTM components, with sigmoid or linear activations assigned by output type.
  • Square loss trains MLP-Frame, MLP-Shot, vsLSTM, and dppLSTM’s first stage, while likelihood trains dppLSTM-single and dppLSTM’s second stage.
  • Training stops after five consecutive validation epochs with descending summarization F-score.
  • The paper frames video summarization as structured prediction requiring sequential modeling of event boundaries, nearby redundancy, and distant visually similar events.
Loading 1605.08110v2…