PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Evaluation of Video Summaries

Mayu Otani, Yuta Nakashima, Esa Rahtu, Janne Heikkilä

arXiv:1903.11328v2cs.CV

TL;DR

Video summarization needs evaluation methods that reliably assess automatically generated skims against human judgments. The paper tests reference-based benchmarks with randomization and proposes correlation-based importance-score evaluation with an accumulative visualization. Random summaries often match or exceed state-of-the-art and sometimes human scores, while segmentation and segment-length distributions strongly influence F1.

  • Problem

    Reference-based video-summary evaluation remains difficult because subjective human judgments and limited annotations constrain reliable assessment.

  • Method

    The paper applies randomization tests to SumMe and TVSum and evaluates importance rankings through correlation with human annotations and accumulative score curves.

  • Results

    Random summaries often achieve state-of-the-art or better performance, sometimes surpassing human scores, while segmentation and segment-length distributions largely determine F1.

  • Takeaways & Limitations

    Importance rankings should be evaluated directly against human-annotated rankings rather than only through final summaries produced by segment selection.

  • Takeaways & Limitations

    The proposed framework covers frame-level importance-score methods and does not address clustering-based approaches or other factors such as comprehensibility, aesthetics, and user-query relevance.

Abstract

from arXiv · show

Video summarization is a technique to create a short skim of the original video while preserving the main stories/content. There exists a substantial interest in automatizing this process due to the rapid growth of the available material. The recent progress has been facilitated by public benchmark datasets, which enable easy and fair comparison of methods. Currently the established evaluation protocol is to compare the generated summary with respect to a set of reference summaries provided by the dataset. In this paper, we will provide in-depth assessment of this pipeline using two popular benchmark datasets. Surprisingly, we observe that randomly generated summaries achieve comparable or better performance to the state-of-the-art. In some cases, the random summaries outperform even the human generated summaries in leave-one-out experiments. Moreover, it turns out that the video segmentation, which is often considered as a fixed pre-processing method, has the most significant impact on the performance measure. Based on our observations, we propose alternative approaches for assessing the importance scores as well as an intuitive visualization of correlation between the estimated scoring and human annotations.

1. Introduction

Video summarization aims to produce short skims preserving important content, but evaluating those summaries remains difficult. This paper tests reference-based evaluation and finds that randomization exposes weaknesses in the protocol, motivating alternative importance-ranking assessments.

  • Motivation: Automatic video summarization produces short skims that preserve important content from rapidly growing video collections.The paper gives sports-event summaries as an example, retaining events such as goals and penalty kicks.
  • Existing pipeline: Modern methods commonly segment videos, predict importance scores, and select segments for the final summary.Importance prediction is described as the most challenging stage, with criteria including visual interestingness, compactness, and diversity.
  • Evaluation problem: Reference-summary evaluation is reusable and reproducible, but its validity remains unresolved despite widespread use of SumMe and TVSum with F1 scores.The datasets provide multiple human-generated summaries or importance scores for each video.
  • Randomization test: Random summaries achieve performance comparable to state-of-the-art methods, revealing critical issues in the current evaluation framework.The randomization test uses random importance scores and segmentation to establish scores achievable by chance.
  • Contributions: The paper reports that F1 is largely determined by video segment-length distributions rather than predicted importance scores.This finding motivates assessing importance rankings directly and visualizing their correlation with human annotations.

2. Related Work

Related work spans importance, compactness, diversity, and learning-based video summarization methods. The section also frames summary evaluation as difficult because subjective quality, limited annotations, and reproducibility constrain existing approaches.

  • Summarization approaches: Video summarization methods target visual interestingness, compactness, representativeness, or diversity when selecting content.Examples include visual dynamics, saliency, redundancy reduction, and diverse event coverage.
  • Learning-based methods: Recent approaches use LSTM models, determinantal point processes, and reinforcement learning to predict importance or select diverse segments.These methods respectively model human importance scores, encourage diverse selection, or optimize frame-selection policies.
  • Common pipeline: Many methods share a pipeline of frame-level scoring, video segmentation, and knapsack-constrained segment selection.The output subset maximizes importance scores under the summary-length constraint.
  • Benchmark findings: Table 1 reports random summaries as comparable to state-of-the-art and human annotations on the SumMe and TVSum benchmarks.It distinguishes average and maximum F1 across reference summaries and includes randomized and leave-one-out human results.
  • Evaluation: Summary evaluation is challenging because quality is subjective, varies across viewers and time, and is amplified by limited videos and annotations.These factors create ambiguity in determining what constitutes a good summary.
  • Evaluation: User studies are costly and difficult to reproduce, while reference-based methods compare generated summaries with human annotations using varied similarity measures.Published alternatives include pixel distance, object counts, precision and recall, and semantic similarity.
  • Benchmarks: SumMe and TVSum are prominent reference-based benchmarks providing human summaries or shot-level importance scores, commonly evaluated with F1.SumMe contains personal videos with annotations from 15–18 annotators, whereas TVSum supplies scores for YouTube videos.

3. Current evaluation framework

The framework evaluates summaries against human references using F1, but the datasets provide references differently: SumMe supplies summaries, while TVSum supplies segment-level importance scores from which references are generated.

  • SumMe: SumMe contains 25 videos with 15–18 human reference summaries per video, each shorter than 15% of the original.
  • TVSum: TVSum provides human importance scores for every two-second interval instead of reference summaries.
  • TVSum: TVSum references are generated by averaging scores within matching segments, then selecting a fixed-length subset maximizing total importance.
  • Evaluation measure: Evaluation computes F1 between predicted and reference frame labels using precision and recall.
  • Evaluation measure: Scores are summarized either by averaging across references or selecting the maximum score for each video.

4. Randomization test

The randomization test evaluates how segmentation and importance scoring contribute to reference-based video-summary scores. Across SumMe and TVSum, segmentation strongly shapes performance, allowing random importance scores to approach human and state-of-the-art results.

  • Randomization test: The test generates content-independent summaries using random importance scores and random segment boundaries, with scores serving as chance-achievable baselines.
  • Segmentation methods: KTS detects visual-feature changes for segmentation, while randomized KTS preserves KTS segment-length distribution but shuffles segment ordering.
  • SumMe results: Segmentation clearly dictates performance, with little impact from importance scoring; random KTS scores are comparable to the best published SumMe methods.
  • TVSum results: Random importance scores achieve similar performance to human annotations for most TVSum segmentation methods, and random two-peak segmentation matches content-based KTS.
  • Discussion: Random summaries reach state-of-the-art performance and sometimes surpass human-level scores, especially with two-peak, KTS, and randomized KTS segmentation.
  • Discussion: Dynamic programming rarely selects long segments because shorter combinations can provide at least as much total score, favoring short segments.
  • Discussion: Using summed frame-level scores makes human summaries clearly outperform random ones on TVSum, although segmentation effects remain.

5. Importance score evaluation framework

The paper evaluates importance scores using rank correlations with human annotations and visualizes their agreement through accumulative score curves. These analyses reveal both method differences and substantial inconsistency among human annotations.

  • Rank order statistics: The framework ranks frames by predicted and human-annotated importance scores, compares the rankings, and averages results across annotators.It uses Kendall’s τ and Spearman’s ρ correlation coefficients.
  • Rank order statistics: Random scoring is evaluated alongside dppLSTM and DR-DSN as a sanity-check baseline expected to produce zero average correlation.The random baseline uses 100 uniformly distributed random sequences for each video.
  • Rank order statistics: Human annotations correlate more strongly than tested methods, while dppLSTM outperforms DR-DSN because it is trained to predict human importance scores.DR-DSN instead aims to maximize diversity.
  • Relation to summary quality: Summaries generated from importance scores positively correlated with human annotations outperform summaries generated from negatively correlated scores in human evaluation.The comparison links importance-score correlation with the quality of the resulting summaries.
  • Annotation inconsistency: Human annotations remain inconsistent because importance is subjective and multiple video parts may express the same content through different viewpoints or expressions.Even the highest human-annotation correlation remains relatively low in absolute value.
  • Accumulative score curves: The accumulative score curve sorts frames by predicted importance and accumulates their averaged human scores, producing a monotonically increasing curve.A rapidly increasing curve indicates high correlation between predicted and human scores.
  • Accumulative score curves: Correlation curves distinguish method behavior from annotation disagreement: tested methods can lie above a random baseline, while some annotators fall below it after labeling nearly opposite content.Figure 8 compares annotator curves, a random expectation, dppLSTM, and DR-DSN.

6. Conclusion

The conclusion finds severe problems in F1-based benchmark evaluation: random summaries often match or exceed state-of-the-art scores, while segmentation and segment-length distributions can dominate the result. It proposes correlation-based importance-score evaluation with accumulative score visualizations, but limits this framework to frame-level importance methods and human-annotation correlation.

  • Conclusion: The current F1-based evaluation framework on widely used video summarization benchmarks has severe problems.The paper assesses the validity of two widely used benchmarks.
  • Conclusion: Randomly generated summaries often achieve similar or better scores than state-of-the-art methods, sometimes surpassing human annotators.The analysis attributes this behavior mainly to video segmentation and segment-length distributions.
  • Proposed evaluation: The paper proposes evaluating predicted importance scores through correlation with human annotations instead of evaluating only the final segment-selected summary.The alternative evaluation is intended to expose importance-score behavior.
  • Proposed evaluation: Accumulative score curves provide an intuitive visualization of importance-score quality relative to multiple human annotations.The curves visualize the correlations produced by the alternative evaluation.
  • Limitations: The proposed framework covers only methods that estimate frame-level importance scores and does not apply to clustering-based segment-selection methods.The paper also identifies story comprehensibility, visual aesthetics, user-query relevance, and larger datasets as future evaluation or research needs.

Distribution of Segment Length

Segment selection produces summaries composed mainly of short segments, with similar segment-length distributions for DR-DSN and dppLSTM. This suggests that the importance-score methods have little effect on the selected segment lengths.

  • Segment-length distribution: Segment selection produces summaries composed of only short segments, while the remaining segments have a different length distribution.Figure 10 compares selected summary segments with remaining segments using orange and blue box plots.
  • Segment-length distribution: DR-DSN and dppLSTM exhibit similar segment-length distributions, indicating that their importance-score prediction methods hardly affect segment lengths.The segment boundaries are generated by two-peak segmentation.

Examples of Video Summaries

Examples produced with KTS segmentation show similar video-summary outputs across two recent methods and the randomized method, regardless of the importance scores.

  • Summary examples: All compared methods produce similar video summaries when using KTS segmentation, despite differences in their importance scores.The comparison includes two recent methods and the randomized method.

Visualization of Importance Score Correlation

Correlation curves compare human annotator agreement with random expectations and learned importance scores on TVSum. Most examples show positive correlation among human annotations.

  • Most examples show positive correlation among human annotators on TVSum.The curves compare each annotator’s correlation with a random-score expectation and results from dppLSTM and DR-DSN.

Results with Different Summary Length

Across summary-length constraints, randomized summaries often match human-annotation performance, while longer summaries tend to receive higher F1 scores. Human evaluations show close comparisons between random and learned scoring, but diverse preferences relative to human summaries.

  • Results with Different Summary Length: For 15%, 25%, and 35% length constraints, randomized summaries matched human annotations with two-peak, KTS, and randomized KTS segmentation.The results used 10 summaries per method and showed similar trends across all three constraints.
  • Results with Different Summary Length: F1 scores tended to increase as the summary-length constraint grew.This trend was observed across the evaluated summary-length settings.
  • Human Evaluation: Random scoring with KTS segmentation received a slightly higher human-evaluation score than DR-DSN scoring, while 46% of answers judged them equally good or bad.Both methods used KTS segmentation, supporting the finding that importance scoring had limited effect under the same segmentation approach.
  • Human Evaluation: Human summaries were slightly preferred over random-scoring summaries, with an averaged comparison score of -0.17.Preferences varied widely, with most videos receiving both strong positive and strong negative judgments from different subjects.
  • Visual Comparisons: The figures compare importance scores and selected frames across methods using shared KTS boundaries, including random, learned, and human-summary conditions.These visualizations isolate differences in scoring or segmentation when comparing generated summaries.
Loading 1903.11328v2…