PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring global diverse attention via pairwise temporal relation for video summarization

Ping Li, Qinghao Ye, Luming Zhang, Li Yuan, Xianghua Xu, Ling Shao

arXiv:2009.10942v1cs.CVcs.MM

TL;DR

Video summarization needs to select informative frames, but recurrent approaches can have difficulty diversifying summaries and incur computational costs. SUM-GDA applies global pairwise attention to video frames, and the authors report stronger results than competing methods across supervised and unsupervised settings. The approach also supports parallel computation at reduced computational cost.

  • Problem

    Existing recurrent video-summarization systems do not explicitly diversify summary frames and require intensive computation, while recurrent models also face long-range dependency and parallelization challenges.

    Source: §I · p. 1

  • Method

    SUM-GDA uses global diverse attention over pairwise temporal relations to estimate frame importance and promote diversity in a convolutional video-summarization model.

    Source: §I · pp. 1–2

  • Results

    SUM-GDA outperforms most competing models by at least 2.8% on SumMe, with gains of 3.1%, 3.3%, and 2.8% in canonical, augmented, and transfer settings, respectively.

    Source: E. Quantitative Results · p. 7

  • Takeaways & Limitations

    The authors report that global diverse attention benefits the diversification of selected key frames, and that the models can run in parallel with significantly less computational cost.

    Source: §I · p. 2, §V · p. 11

Abstract

from arXiv · show

Video summarization is an effective way to facilitate video searching and browsing. Most of existing systems employ encoder-decoder based recurrent neural networks, which fail to explicitly diversify the system-generated summary frames while requiring intensive computations. In this paper, we propose an efficient convolutional neural network architecture for video SUMmarization via Global Diverse Attention called SUM-GDA, which adapts attention mechanism in a global perspective to consider pairwise temporal relations of video frames. Particularly, the GDA module has two advantages: 1) it models the relations within paired frames as well as the relations among all pairs, thus capturing the global attention across all frames of one video; 2) it reflects the importance of each frame to the whole video, leading to diverse attention on these frames. Thus, SUM-GDA is beneficial for generating diverse frames to form satisfactory video summary. Extensive experiments on three data sets, i.e., SumMe, TVSum, and VTW, have demonstrated that SUM-GDA and its extension outperform other competing state-of-the-art methods with remarkable improvements. In addition, the proposed models can be run in parallel with significantly less computational costs, which helps the deployment in highly demanding applications.

I. INTRODUCTION

Video summarization must select informative, diverse frames, while prior recurrent and hand-crafted approaches have limitations in efficiency and modeling complex temporal structure. SUM-GDA uses global pairwise attention to score frames, and the paper reports improved results over competing methods.

I. INTRODUCTION

  • Hand-crafted-feature methods are not trained end-to-end efficiently and may miss high-level semantics in videos with diverse, complex scenes.Recurrent models also face limited GPU parallelization, long-range dependency decay, and semantic discontinuity challenges.Source: §I · p. 1
  • Global diverse attention uses pairwise information to quantify frame importance while promoting diversity among selected frames.The model aggregates attention across the video and gives greater emphasis to frame pairs with higher dissimilarity.Source: §I · pp. 1–2
  • SUM-GDA outperforms state-of-the-art approaches in supervised and unsupervised settings across the evaluated datasets.The introduction identifies SumMe, TVSum, and VTW as evaluation datasets.Source: §I · p. 2
  • SUM-GDA models temporal dependencies through pairwise relations between every two frames, addressing the long-range dependency problem associated with RNNs.The pairwise similarity matrix is computed directly without processing intermediate frames, at limited computational cost.Source: §I · pp. 1–2
  • The paper explores supervised, unsupervised, and semi-supervised settings and investigates summary diversity and optical-flow features.It also reports quantitative and qualitative studies.Source: §I · p. 2

II. RELATED WORK

Related work spans heuristic unsupervised and learned supervised summarization, including recurrent deep models and self-attention. SUM-GDA adapts attention to video frames and uses convolutional features with global pairwise temporal relations.

II. RELATED WORK

  • Prior work includes unsupervised clustering and dictionary-learning methods, supervised frame scoring, and deep models using recurrent networks, reinforcement learning, or convolutional sequences.These approaches pursue key-shot selection through criteria such as relevance, representativeness, diversity, or learned importance.Source: §II · pp. 2–3
  • Earlier self-attention for summarization measured frame importance but neglected frame diversity; SUM-GDA uses pairwise temporal relations to address both.The related-work discussion identifies Fajtl et al. as an example of importance-only self-attention.Source: §II · p. 3

III. THE PROPOSED APPROACH

  • SUM-GDA uses pre-trained CNN features and a pairwise similarity matrix to encode temporal relations across frames at different stride lengths.Its diverse attention weights are transformed into frame-importance scores, and the dissimilarity representation supports parallel computation across multiple GPUs.Source: §III · p. 3

A. Global Diverse Attention

Global diverse attention aggregates pairwise frame relations to assess each frame against the whole video. It then derives dissimilarity-based weights to support diverse video summaries.

A. Global Diverse Attention

  • Global diverse attention aggregates information across frame pairs to evaluate each frame's importance to the whole video and derive pairwise dissimilarity scores.The scores indicate appearance variation between frames.Source: A. Global Diverse Attention · p. 3
  • The pairwise attention matrix encodes temporal relations between frame features; learned projections and a scaling factor produce each paired-frame attention value.The text sets q = D and notes scaling is used because unscaled attention can produce very small gradients after softmax.Source: A. Global Diverse Attention · p. 3
  • Normalized pairwise attention weights are converted into a dissimilarity vector so the model can attend to frames dissimilar from others across the video.The resulting weights indicate how each frame differs from the whole input video and suggest frame diversity.Source: A. Global Diverse Attention · p. 4
  • The normalized dissimilarity representation is linearly mapped into weighted context vectors that capture frame context.The context vectors use a learned projection and element-wise product.Source: A. Global Diverse Attention · p. 4

B. The SUM-GDA Model

SUM-GDA uses a CNN-based score-regression architecture with a DPP component to diversify selected frame subsets. Its kernel combines frame scores with pairwise feature similarity, and supervised training incorporates the proposed loss functions.

B. The SUM-GDA Model

  • DPP assigns low probability to subsets containing similar frames, favoring subsets with greater variation or diversity.The proposed kernel models quality-diversity using frame scores and pairwise similarity between transformed frame features.Source: B. The SUM-GDA Model · p. 4
  • The DPP kernel represents pairwise frame similarity using transformed features and weights it by the two frames’ scores.The kernel is introduced as an enhancement inspired by quality-diversity decomposition.Source: B. The SUM-GDA Model · p. 4
  • In supervised training, ground-truth key-frame annotations are used to define a key-frame loss.The supervised loss combines the referenced losses, and the model optimizes its loss functions iteratively.Source: B. The SUM-GDA Model · p. 4
  • SUM-GDA combines DPP with score regression in an end-to-end neural architecture trained by iteratively optimizing its loss functions.The model uses a score-regression module and linear embedding module after the feed-forward layer.Source: B. The SUM-GDA Model · p. 4

C. Unsupervised SUM-GDA

Unsupervised SUM-GDA learns from untrimmed videos without ground-truth supervision. It replaces key-frame loss with length regularization and uses a repelling loss to encourage diversity among frames.

C. Unsupervised SUM-GDA

  • SUM-GDA_unsup learns from untrimmed videos without supervision or ground-truth annotations.The paper notes that collecting such annotations requires expensive human labeling.Source: C. Unsupervised SUM-GDA · pp. 4–5
  • The unsupervised variant replaces key-frame loss with length regularization balanced by the summary ratio.The change addresses disagreement among users asked to evaluate the same video’s summary.Source: C. Unsupervised SUM-GDA · p. 4
  • Repelling loss measures mean pairwise similarity across all video frames, with lower values indicating a more diverse subset.It is used to enhance diversity when summary annotations are unavailable.Source: C. Unsupervised SUM-GDA · p. 5

D. Summary Generation

Summary generation segments a video into shots, constrains the summary length, and selects shots using their model scores under a 0/1 Knapsack formulation.

D. Summary Generation

  • Kernel Temporal Segmentation partitions the test video into shot segments before selection.The segmentation generates change points.Source: D. Summary Generation · p. 5
  • The summary length is constrained to the proportion of user summary length to original video length.Source: D. Summary Generation · p. 5
  • The 0/1 Knapsack algorithm selects shots using their mean scores and shot lengths to form the final summary.Selected shots are those whose selection indicator is one.Source: D. Summary Generation · p. 5

E. Summary Diversity Metric

The proposed summary diversity metric evaluates selected shots by their distance from nearby selected-shot cluster centers. Smaller metric values indicate greater diversity.

E. Summary Diversity Metric

  • The paper proposes summary diversity metric ζ because F-score does not reflect diversity within generated summaries.F-score measures overlap between generated and user summaries.Source: E. Summary Diversity Metric · p. 5
  • Metric ζ averages the Euclidean distance from each selected key shot to its nearest selected-shot cluster center.The shot feature vector averages frame features across that shot.Source: E. Summary Diversity Metric · p. 5
  • Smaller ζ indicates greater diversity; the paper treats selected key shots as cluster centers and describes them as usually more representative than unselected shots.Source: E. Summary Diversity Metric · pp. 5–6

IV. EXPERIMENTS

The experiments evaluate SUM-GDA on SumMe, TVSum, and VTW using user-summary agreement, across canonical, augmented, and transfer settings where applicable.

A. Data Sets

  • SumMe has 25 user videos, TVSum has 50 videos annotated by 20 users, and VTW has 2,529 YouTube videos with highlighted key shots.OVP and YouTube are also used to augment and transfer training data.Source: A. Data Sets · p. 6

B. Evaluation Metric

  • F-score measures agreement between generated and user-annotated summaries through the harmonic mean of precision and recall over overlapping key shots.Source: B. Evaluation Metric · p. 6

C. Evaluation Settings

  • SumMe and TVSum use five-fold cross-validation with 80% training and 20% testing; VTW uses 2,000 training videos and the remaining videos for testing.For SumMe, the maximum F-score across reference summaries is reported; for TVSum, scores are averaged across summaries.Source: C. Evaluation Settings · p. 6

IV. EXPERIMENTS

D. Implementation Details

SUM-GDA generally outperforms competing methods across the reported settings, while the unsupervised variant performs especially well on TVSum and transfer-setting SumMe. The experiments also examine sensitivity to summary ratio.

D. Implementation Details

  • Implementation uses 1,024-dimensional pre-trained GoogLeNet pool5 frame features, 1,024 hidden units, dropout 0.6, and L2 weight decay 10^-5.Training uses Adam for up to 200 epochs; SUM-GDAunsup uses σ = 0.3.Source: D. Implementation Details · p. 6

E. Quantitative Results

  • The comparisons cover supervised and unsupervised methods on SumMe and TVSum, and supervised methods on VTW.The cited comparison tables report F-score (%) and the VTW comparison is limited to supervised methods.Source: E. Quantitative Results · pp. 6–7, Table IV · p. 7
  • 3.1%, 3.3%, and 2.8% are SUM-GDA’s reported gains over other methods on SumMe in canonical, augmented, and transfer settings, respectively.SUM-GDAunsup is 1.2% above the second-best method on SumMe transfer and achieves top performance on TVSum in canonical and augmented settings.Source: E. Quantitative Results · p. 7
  • The reported VTW comparison is an F-score (%) comparison, and SUM-GDA is described as obtaining larger gains there than other methods.The authors attribute the gains to using temporal relations among all frame pairs to encode frame importance.Source: E. Quantitative Results · p. 7, Table IV · p. 7
  • F-score degrades sharply when the summary ratio σ is too large or too small; the best reported σ is 0.3 generally and 0.4 on VTW.The authors relate VTW’s robustness to variation in its summary proportions and note that 0.4 is near their mean.Source: E. Quantitative Results · p. 7, Figure 3 · p. 7

F. Computational Issues

The paper reports that SUM-GDA variants are more efficient than competing methods and describes how parallel computation and loss terms relate to runtime and summary diversity. Qualitative examples illustrate frame selection and attention weights.

F. Computational Issues

  • SUM-GDA and SUM-GDAunsup are reported as much more efficient, with SUM-GDAunsup reducing training and test time while offering a better efficiency–performance trade-off.Figure 4 compares training time, test time, and F-score on SumMe and TVSum in the canonical setting.Source: F. Computational Issues · p. 7
  • GDA requires O(n^2d) time versus O(nd^2) for RNNs, and uses O(1) sequential operations versus O(n), when enough GPUs are available.The authors state GDA is faster when sequence length n is smaller than feature dimension d, as is usually the case for the tested benchmarks.Source: F. Computational Issues · p. 8

G. Ablation Study

  • Removing length regularization deteriorates performance, while adding repelling loss improves F-score by 1.1% on SumMe and 0.7% on TVSum versus without it.The authors say repelling reduces pairwise similarity among partial frames, enhancing global diverse attention.Source: G. Ablation Study · p. 8, Table V · p. 8
  • Variation loss improves F-score by 1.2% on SumMe and 0.8% on TVSum, and the authors report improvements from adopting GDA in both model variants.Source: G. Ablation Study · p. 8, Table V · p. 8

H. Qualitative Results

  • SUM-GDA selects key shots around frame-score peaks using the 0/1 Knapsack algorithm, while SUM-GDAunsup produces sparser frame scores.The authors attribute sparsity to constraining the mean frame score near the summary ratio.Source: H. Qualitative Results · p. 8
  • Visually similar frames in a traffic-accident example receive different global diverse attention weights, consistent with selecting only one redundant frame for the summary.Source: H. Qualitative Results · p. 8

I. Semi-Supervised Scenario

The semi-supervised experiments report improved performance as unlabeled training data increases, while SUM-GDA also generates more diverse summaries on SumMe and TVSum. The visualizations describe how selected key shots represent nearby shots while remaining separated across groups.

I. Semi-Supervised Scenario

  • Summarization performance consistently improves as the number of unlabeled samples increases in the SumMe and TVSum semi-supervised experiments.The setup uses unlabeled videos sampled from VTW; test videos are used only for testing.Source: §I · p. 8, Table VI · p. 9

J. Diversity of Generated Summaries

K. Optical Flow Features

The optical-flow experiments show dataset-dependent effects: flow improves results on TVSum but lowers performance on SumMe and VTW. The conclusion reports stronger summaries than several state-of-the-art approaches, while noting limitations in optical-flow performance and efficiency under dense sampling.

K. Optical Flow Features

  • Optical flow greatly improves performance on TVSum but lowers it on SumMe and VTW, where still backgrounds may matter alongside motion.The experiments use RGB, optical-flow, and combined inputs under the Canonical setting.Source: K. Optical Flow Features · pp. 9, 11, Table VIII · p. 11

V. CONCLUSION AND FUTURE WORK

  • SUM-GDA and SUM-GDAunsup yield more promising summaries than several state-of-the-art approaches, and the reported visualizations suggest global diverse attention benefits key-frame diversity.The paper also suggests global diverse attention might help train base learners for deep ensemble learning.Source: §V · p. 11
  • The efficiency advantage is stated for sequence lengths below the embedding dimension; denser sampling may require large computations.The paper identifies further efficiency improvements as a direction for wider scenarios.Source: §V · p. 11
  • Optical flow does not always improve performance and can sometimes reduce summarization quality.The authors propose improving optical-flow extraction and exploring other ways to capture video temporal structure.Source: §V · p. 11
Loading 2009.10942v1…