PaperlayerView exact revision on arXiv

Source-linked AI summary

Summary Transfer: Exemplar-based Subset Selection for Video Summarization

Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman

arXiv:1603.03369v3cs.CV

TL;DR

The paper addresses how to summarize growing video collections when frame decisions are interdependent and existing methods lack learned global subset-selection criteria. It transfers summary structures nonparametrically from annotated exemplar videos to unseen videos, extending the approach with semantic side information and subshot selection. Across several benchmarks, it reports promising results compared with existing techniques, while noting limitations when test-video relationships are mediated through unreliable training neighbors.

  • Problem

    Video summarization requires interdependent subset decisions, but existing methods lack knowledge of the global criteria humans use to create summaries.

  • Method

    The method nonparametrically transfers summary structures from annotated exemplar videos to novel videos, with extensions using semantic categories and subshot-based representations.

  • Results

    The method achieves promising results on several benchmark datasets compared with an array of nine existing techniques.

  • Takeaways & Limitations

    Summary structures learned from exemplar videos can guide supervised keyframe or subshot selection for new videos, including videos that are not closely visually similar.

  • Takeaways & Limitations

    The test-video kernel does not directly model relationships among the test video’s own frames, so transfer can fail when the closest training neighbor is semantically irrelevant.

Abstract

from arXiv · show

Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections. We propose a novel subset selection technique that leverages supervision in the form of human-created summaries to perform automatic keyframe-based video summarization. The main idea is to nonparametrically transfer summary structures from annotated videos to unseen test videos. We show how to extend our method to exploit semantic side information about the video's category/genre to guide the transfer process by those training videos semantically consistent with the test input. We also show how to generalize our method to subshot-based summarization, which not only reduces computational costs but also provides more flexible ways of defining visual similarity across subshots spanning several frames. We conduct extensive evaluation on several benchmarks and demonstrate promising results, outperforming existing methods in several settings.

1. Introduction

Video summarization is a structured subset-selection problem whose interdependent decisions and unknown global human criteria challenge existing methods. The paper proposes nonparametric supervised transfer of summary structures from exemplar videos and reports stronger benchmark performance than competing approaches.

  • Problem: Video summarization selects interdependent subsets of frames or subshots rather than independent binary labels.The paper focuses on keyframe or subshot selection, where the output is an ordered subset of the video sequence.
  • Problem: Existing methods use heuristics such as representativeness, diversity, interestingness, and relevance, but optimizing their combination remains unresolved.Human annotators’ global criteria for manually creating summaries are largely unknown.
  • Approach: Nonparametric supervised learning transfers summary structures from annotated exemplar videos to novel input videos.The approach is motivated by the observation that similar videos tend to share similar summary structures.
  • Approach: The method learns from human-created summaries to select subsets directly, rather than transferring simple frame-level relevance labels.Kernel-based representations encode frames or subshots and their subset structures for structured transfer.
  • Evaluation: The paper presents an extensive empirical study reporting better summarization results than competing methods across several benchmark datasets.It describes the study as validating the proposed supervised method and advancing supervised video summarization.

2. Related Work

Earlier video summarization methods largely rely on visual heuristics and are mostly unsupervised, while subset-selection formulations enable supervision from human summaries. This paper differs by transferring summary structures nonparametrically from exemplar training videos to test videos, using a conceptual pipeline that combines frame similarities with structured decoding.

  • Existing methods: Prior methods compute frame-level visual features and apply criteria such as representativeness, diversity, saliency, relevance, or anomaly detection.Keyframe methods may use optical flow, image differences, object tracks, or important-object information; skimming methods select representative subshots.
  • Proposed approach: The conceptual pipeline computes training-test frame similarity, encodes training summary structures as kernel matrices, combines them into a test-video kernel, and decodes a globally optimal frame subset with a DPP.Figure 1 presents this structure-transfer process from human-created training summaries to a new video.
  • Existing methods: Most existing techniques are unsupervised and do not learn the subset-selection procedure itself.Some supervised components learn saliency, viewpoints, or event fragments without learning the complete subset-selection process.
  • Supervised subset selection: Subset-selection formulations enable supervised learning from human-created summaries through global objectives or probabilistic models over ground-truth subsets.Prior examples include a submodular objective and a probabilistic model maximizing the probability of ground-truth subsets.
  • Proposed approach: The proposed approach transfers summary structures nonparametrically from exemplar training videos to test videos, unlike previous parametric models.It generalizes locally by directly exploiting patterns in training data when an annotated video lies in the test video’s neighborhood.

3. Approach

The approach casts video summarization as structured subset selection and transfers summary structures from annotated exemplar videos to unseen inputs. It uses DPP-based kernels, visual alignment, and extensions for semantic side information and subshot selection.

  • Core formulation: The method learns subset-selection criteria from videos with human-created summaries and applies them to unseen videos.The ground set is the video's frames, and the output is an ordered subset.
  • Core formulation: DPPs model ordered subsets while capturing dependencies among selected frames and remaining computationally tractable.The DPP mode provides the most probable subset, although exact MAP inference is NP-hard.
  • Kernel modeling: The summarization kernel defines the desired subset through a bivariate function over frame features, with φ(·) providing the feature mapping.Possible choices include identity, Gaussian RBF, or neural-network mappings.
  • Non-parametric transfer: Instead of discarding training data after learning one φ(·), the method constructs each test video's kernel by comparing it with annotated videos and their summaries.This combines visual similarity between videos with the summary structures observed in training exemplars.
  • Extensions: Category or genre information guides transfer toward semantically consistent training videos, while subshot selection supports contiguous event-level summaries.Subshot-based transfer also reduces computational costs and permits more flexible similarity across multiple frames.

4. Experiment

The experiments evaluate nonparametric summary transfer across five benchmark datasets, including category-aware and subshot-based variants. The method generally outperforms competing approaches, while qualitative analysis identifies a failure mode caused by semantically mismatched neighbors.

  • Experimental setup: Experiments use five benchmark datasets with 80% of videos for training and 20% for testing, repeated across rounds for averaged performance and standard errors.Keyframe experiments cover OVP, YouTube, and Kodak; subshot experiments cover MED, SumMe, and YouTube.
  • Main results: On four of five datasets, the nonparametric method achieves the best results, including datasets with weak visual similarity such as SumMe.The authors evaluate against supervised and unsupervised methods using F-score, precision, and recall.
  • Category information: Category-aware transfer improves summarization by using semantically consistent training videos, including independently generated categories for SumMe.On YouTube, semantically close videos allow the nonparametric approach to outperform a parametric approach with the same or more annotated data.
  • Subshot-based summarization: Subshot-based summarization improves over frame-based summarization, especially when no category prior is available, while also reducing computational cost.The authors attribute the improvement to more robust subshot-level similarity measures.
  • Qualitative analysis: A failure case occurs when visual similarity selects a semantically mismatched neighbor, causing the method to miss the final two ground-truth frames.The authors identify indirect interaction among test-video frames through training videos as a weakness and suggest a hybrid parametric–nonparametric direction.

5. Conclusion

The paper presents a nonparametric supervised method that transfers summary structures from training videos to test videos, with semantic-category and subshot extensions. It reports promising benchmark results against nine existing techniques.

  • The method nonparametrically transfers summary structures from training videos and their summaries to unseen test videos.It is designed for supervised video summarization of structured subsets.
  • Semantic side information, including video categories, can guide transfer toward semantically consistent training videos.
  • The method achieves promising results on several benchmark datasets compared with an array of nine existing techniques.
  • The approach extends to subshot-based summarization by segmenting training videos into smaller chunks and learning parameters for each segment.Each chunk is treated as a separate training video containing part of the original summary.
  • For sequential modeling, test videos are segmented temporally and summarized using ground sets that combine the current segment with the previously selected subset.

B. Datasets

The evaluation uses five benchmark datasets, including keyframe-based Kodak, OVP, and YouTube collections with human-created frame-level summaries. Their sizes and sampling procedures define the experimental ground sets.

  • The study validates its approach on five benchmark datasets.
  • The OVP, YouTube, and Kodak datasets contain 50, 39, and 18 videos, respectively.OVP and YouTube provide five human-created frame-level summaries per video, whereas Kodak provides one per video.
  • For keyframe summarization, frames are sampled at one frame per second for OVP and YouTube and two frames per second for Kodak.Transition and near-monotone frames are pruned; the resulting ground sets average 84, 128, and 50 frames for OVP, YouTube, and Kodak.

B.2. SumMe dataset

SumMe contains heterogeneous videos summarized by many users, so the study performs subshot-based summarization and investigates whether synthetic category information helps transfer summary structure.

  • The SumMe dataset contains 25 videos averaging 2m40s, each cut and summarized by 15 to 18 people.
  • Because the average ground-truth shot-based summary is 13.1% of the original video, the study performs subshot-based summarization.
  • SumMe videos are heterogeneous and lack predefined categories, although some videos have varying degrees of relatedness.
  • The study creates two synthetic super-categories by collapsing ten TVSum3 video categories into partitions based largely on crowd presence.Super-category I emphasizes raw videos with crowds, while Super-category II contains the remaining activity types.
  • Oracle summaries are constructed by scoring frames across user summaries, averaging scores within KTS shots, and selecting top shots totaling around 15% of the video.

B.3. MED dataset

The MED evaluation uses annotated videos from a large event dataset but relies on pre-segmented shots and precomputed features, requiring shot-level oracle construction for subshot summarization.

  • MED contains 12,249 videos, including 2,389 videos across ten event categories and 9,860 videos labeled null.Only 160 videos from the ten specified categories are annotated with summaries.
  • The study performs subshot-based summarization on MED.
  • The dataset lacks original videos and instead provides previously segmented shot boundaries and precomputed Fisher vectors.It averages 27 shots per video, and its shot boundaries often differ from those determined by human annotators.
  • Training oracle summaries are created by mapping user annotations onto predetermined shots and averaging frame importances within each shot.
  • For MED, the experiments use the Fisher vectors supplied with the dataset because raw video was not provided.

C. Evaluation protocols

The evaluation compares automatic summaries with human-created summaries using matched-frame precision, recall, and F-score, under dataset-specific length and aggregation protocols. Experiments also assess visual-similarity choices.

  • Metrics: F-score, precision, and recall are computed from matched frames between automatic and human-created summaries.Frames qualify as matched when their visual difference is below a threshold, with one-to-one matching enforced.
  • Metrics: For multiple human summaries, metrics are averaged or maximized across annotators and then averaged across videos.The paper follows the maximum-over-users protocol for the cited comparison setting.
  • Length protocol: Summary lengths are constrained to at most 15% of video duration, with excess selected shots ranked by diagonal L-kernel values.Higher diagonal values indicate more important items in the subset-selection framework.
  • Length protocol: For MED, subshots are treated as frames, and KVS outputs are evaluated using the top 15% or 30% of shots.User annotations are aligned with predetermined shot boundaries before evaluation.
  • Similarity analysis: Nonlinear Gaussian-RBF similarity generally outperforms linear similarity, while learning a non-identity metric improves performance only marginally.The experiments therefore use sim2 in subsequent evaluations.

D.2. Results on the complete YouTube dataset

The complete YouTube dataset contains 39 non-cartoon videos, including eight outside the Sports and News categories. Results on all 39 videos are provided for comparability with prior work.

  • Dataset: The complete YouTube dataset contains 39 videos after excluding cartoon videos, with eight videos neither Sports nor News.The main-text experiments use 31 videos to compare category-prior settings.

D.3. Detailed results with category prior

Category information improves summary transfer across most reported settings, including when categories must be predicted rather than supplied as ground truth. The experiments span YouTube, SumMe, and MED.

  • Category prior: Both soft and hard category-specific transfer outperform category-agnostic transfer in most cases.The comparison is reported across category-prior experiments on the evaluated datasets.
  • Category prior: Videos with related semantic content can share summary structures even when their visual settings differ.The paper gives birthday-party and outdoor-activity videos as an example of potentially useful structural overlap.
  • Category prior: Category priors can still improve performance when test-video categories are inferred by a classifier.The benefit is not restricted to experiments with known ground-truth categories.
  • Feature comparison: Deep recognition features do not provide much benefit over shallow features for summarization in the reported comparison.The study compares Fisher vectors and color histograms with GoogLeNet CNN features.

D.5. Comparison to seqDPP

The proposed method outperforms seqDPP on Kodak and YouTube, while qualitative results attribute its advantage to supervised representativeness and nonparametric transfer. Its main trade-offs are higher test-time cost and sensitivity to visual similarity.

  • Quantitative comparison: The proposed method outperforms seqDPP on Kodak and YouTube, whose contents are more diverse and challenging than OVP.The paper suggests seqDPP may perform better on less redundant edited videos because of its higher precision.
  • Computational comparison: Testing takes about 1 second per YouTube video, compared with 0.5 seconds for seqDPP, but training uses fewer parameters and less time per configuration.The proposed model uses about 9,000 parameters versus 80,000 for seqDPP; training takes about 1 minute versus 9 minutes on YouTube.
  • Qualitative comparison: Human-like summaries are attributed to supervised identification of representative content and transfer of summarization criteria through kernel matrices.The transferred kernels remove several uninformative frames, improving precision relative to seqDPP in the qualitative analysis.
  • Failure analysis: The qualitative failure case points toward jointly combining the test-video kernel with the transferred kernel to recover missing frames and improve recall.The paper associates the failure with shorter summaries and missing frames.
  • Efficiency: Subshot-based transfer and down-sampling reduce computation, while hard category-specific transfer enables transfer from fewer training videos with improved performance.Subshot similarity reduces matrix size without worsening performance in the cited setting.
  • Failure analysis: Summarization quality is positively correlated with visual similarity between testing and annotated videos.The paper proposes similarity thresholding as a preliminary fail-safe mechanism for detecting likely failures.
Loading 1603.03369v3…