Source-linked AI summary
A General Framework for Edited Video and Raw Video Summarization
Xuelong Li, Bin Zhao, Xiaoqiang Lu
TL;DR
Existing video-summarization approaches often target one video type, and scarce training data makes supervised property-weight learning difficult. This paper combines four property models in a general framework with separate learned weights and mixed training data. Experiments report that the models are necessary for both video types and that mixed training helps only when mixing-coefficients are used.
Problem
Existing raw-video property models may not generalize to edited videos, while scarce training data makes supervised property-weight learning difficult.
Source: §I · p. 2Method
The framework combines four summary-property models with separately supervised property-weights and a mixed edited/raw training set using per-video mixing-coefficients.
Source: §I · p. 2Results
Across edited- and raw-video tests, all four models were necessary, separate property-weights were more reasonable, and mixed training helped only with mixing-coefficients.
Source: E. Discussion · p. 11Takeaways & Limitations
The framework provides a shared approach to edited- and raw-video summarization while using combined training data to address training-data scarcity.
Source: §I · p. 2
Abstract
from arXiv · showhide
In this paper, we build a general summarization framework for both of edited video and raw video summarization. Overall, our work can be divided into three folds: 1) Four models are designed to capture the properties of video summaries, i.e., containing important people and objects (importance), representative to the video content (representativeness), no similar key-shots (diversity) and smoothness of the storyline (storyness). Specifically, these models are applicable to both edited videos and raw videos. 2) A comprehensive score function is built with the weighted combination of the aforementioned four models. Note that the weights of the four models in the score function, denoted as property-weight, are learned in a supervised manner. Besides, the property-weights are learned for edited videos and raw videos, respectively. 3) The training set is constructed with both edited videos and raw videos in order to make up the lack of training data. Particularly, each training video is equipped with a pair of mixing-coefficients which can reduce the structure mess in the training set caused by the rough mixture. We test our framework on three datasets, including edited videos, short raw videos and long raw videos. Experimental results have verified the effectiveness of the proposed framework.
I. INTRODUCTION
Edited and raw videos differ in structure but share summary goals: preserving important content, representing the video, avoiding redundancy, and maintaining a smooth storyline. The paper proposes a shared framework with four property models, supervised domain-specific weights, and a mixed training set equipped with mixing-coefficients.
I. INTRODUCTION
- Prior raw-video property models often target specific video kinds and do not readily generalize to edited videos, while scarce data makes supervised weight learning difficult.The paper identifies manual weight setting as one response to this training-data shortage.Source: §I · pp. 1–2
- Four models capture importance, representativeness, diversity, and storyness, and are designed for both edited- and raw-video summarization.The framework aims to account for commonalities and differences between the two video types.Source: §I · p. 2
- A weighted score combines the four models, with property-weights learned under supervision separately for edited and raw videos.The paper argues that separate weights are more reasonable than a common set for both video types.Source: §I · p. 2
- A combined edited-and-raw training set addresses scarce training data, while per-video mixing-coefficients reduce structural interference from the rough mixture.Each video receives a pair of coefficients reflecting its relevance to edited- and raw-video summarization.Source: §I · p. 2
II. RELATED WORKS
Prior work on edited-video summarization emphasizes representative shots, while raw-video approaches often focus on removing unimportant content. The section reviews clustering, dictionary learning, object-based methods, and score-based selection across these settings.
II. RELATED WORKS
- Clustering groups similar frames or shots, then uses cluster centers as representative elements for edited-video summaries.Variants include classic algorithms, sequential initialization, temporal graphs, scene recognition, and co-clustering videos on the same topic.Source: §II · p. 2
- Dictionary-learning methods select a small set of representative frames or shots as summary elements, with sparse coding and locality constraints among the approaches reviewed.The section also notes sequential determinantal point processes as a way to constrain diversity.Source: §II · p. 2
- Raw-video approaches commonly remove shots without significant people or important objects, using object detection and multi-level visual features.The cited examples include hierarchical visual models and removing frames when relevant objects are not detected.Source: §II · p. 2
- Score-based methods rank frames or shots using visual-importance features or combinations including aesthetics and landmarks.The section describes these methods as selecting higher-scoring frames or shots for the summary.Source: §II · p. 2
- Some multi-model approaches target egocentric videos and manually set model weights because training data are scarce; a convolutional neural network is also cited for scoring shots.The paper contrasts these precedents with its general framework for edited and raw videos.Source: §II · p. 3
III. THE GENERAL SUMMARIZATION FRAMEWORK
The framework scores candidate summaries using property models intended to capture important content, video coverage, low redundancy, and storyline smoothness. Its property models adapt these goals to both edited and raw videos.
A. Property models
- The importance model sums frame importance within shots and scores a summary by the importance of its selected shots; edited videos replace the raw-video follow-object feature with bounding-box size and location.The raw-video regression features include aesthetics and quality, detected faces, people and landmarks, and follow-object likelihood.Source: A. Property models · pp. 3–4
- Representativeness measures how well a summary covers video information by reconstructing each shot from nearby key-shots, limiting information loss.The nearest key-shots on the left and right are used to reconstruct a shot; smaller Euclidean distances indicate its content is captured.Source: A. Property models · p. 4
- Diversity penalizes similarity between consecutive selected shots, while allowing similar non-consecutive shots that may preserve distinct information.The model is designed to reduce redundancy while remaining computationally efficient.Source: A. Property models · p. 4
- Storyness is highest when selected key-shots are distributed uniformly across the video, which the paper associates with storyline smoothness and easier understanding.The model is intended to apply to both edited and raw videos, unlike a prior consecutive-object relationship model that does not suit edited sequences.Source: A. Property models · p. 4
B. Score Function
The score function combines four summary properties to rank candidate shot subsets, while its property-weights balance their contributions. Edited and raw videos require different weighting, and mixing-coefficients address the structural mismatch in their combined training set.
B. Score Function
- Overemphasizing importance can fill a summary with shots containing specified important objects while undermining the other properties.The paper motivates a comprehensive score function to balance these competing influences.Source: B. Score Function · pp. 4–5
- The desired summary is the shot subset S that maximizes F(V, S, w), the score reflecting its ability to summarize the video.The summary is selected from subsets of the video's shots.Source: B. Score Function · p. 5
- The score function is a linear combination of importance, representativeness, diversity, and storyness, with property-weight w encoding their relative emphasis.The four properties measure different aspects of summary quality.Source: B. Score Function · p. 5
- Because edited and raw videos differ in structure and in their emphasis on summary properties, a common property-weight is not appropriate for both.Raw videos emphasize importance, whereas edited videos focus more on representativeness.Source: B. Score Function · p. 5
- The combined training set uses mixing-coefficients to reduce interference from structural differences between edited and raw videos.The coefficients reflect each training video's relevance to the two summarization tasks.Source: B. Score Function · p. 5
C. Mixing-coefficient
Each training video receives two mixing-coefficients that express its relevance to edited and raw video summarization. The coefficients are based on similarity between videos' summarization patterns and are used in learning the corresponding property-weights.
C. Mixing-coefficient
- Each training video receives a pair of mixing-coefficients representing its relevance to edited and raw video summarization, respectively.The coefficients are later used to learn the property-weights for both tasks.Source: C. Mixing-coefficient · pp. 5–6
- A video’s summarization pattern p encodes the mapping from its shot features to its summary, with an L1 constraint encouraging sparsity.The features form matrix V and the shot labels y mark key shots as 1 and others as 0.Source: C. Mixing-coefficient · p. 5
- Similarity between two summarization patterns is determined by the inner product of their normalized mapping-vector directions.Closer directions indicate more similar patterns.Source: C. Mixing-coefficient · p. 5
- For video i, each mixing-coefficient is calculated by averaging its pattern similarities with the videos of the corresponding type.The edited-video coefficient averages similarities to edited videos, and the raw-video coefficient averages similarities to raw videos.Source: C. Mixing-coefficient · pp. 5–6
D. Property-weight
The framework learns separate property-weight vectors for edited and raw video summarization. It uses mixing-coefficients to weight training videos in a maximum-margin learning objective with regularization.
D. Property-weight
- Edited and raw video summarization use separate property-weight vectors, we and wr, respectively.The weights are learned individually for each video type.Source: D. Property-weight · p. 6
- The maximum-margin approach learns weights so the ground-truth summary scores above other summary candidates by a margin.The paper uses a structured hinge loss, with task loss counting the percentage of summary shots absent from the ground truth.Source: D. Property-weight · p. 6
- For edited-video weight learning, each training video is weighted by its edited-video mixing-coefficient, and a regularization term smooths the weight distribution.The objective is convex; λ balances the loss and regularization terms and is fixed at 0.01.Source: D. Property-weight · p. 6
- The raw-video weight vector is learned in the same way using the raw-video mixing-coefficient.Source: D. Property-weight · p. 6
E. Optimization
The optimization uses submodularity to avoid evaluating every possible shot subset: diminishing gains make greedy selection applicable. Accelerated Greedy selects summaries, while Projected Subgradient Descent learns non-negative property-weights.
E. Optimization
- Algorithm 1 learns edited- and raw-video property-weights from the training set and returns the two weight vectors.Its procedure applies Projected Subgradient Descent and Accelerated Greedy.Source: E. Optimization · p. 6
- With 2^|V| possible shot subsets, the framework uses submodularity to avoid scoring every subset and ranking them.Submodular functions have diminishing gains as selected elements accumulate.Source: E. Optimization · p. 6
- Accelerated Greedy approximately maximizes the summary score, while Projected Subgradient Descent optimizes the property-weights under a non-negativity constraint.The score is a non-negative weighted sum of submodular property models; the paper reports O(|V|) test-time complexity for the main summary-generation computations.Source: E. Optimization · p. 7
IV. EXPERIMENTAL RESULTS
The framework is evaluated on datasets for edited, short raw, and long raw videos, with comparisons to popular approaches for both video types.
IV. EXPERIMENTAL RESULTS
- The evaluation covers TVsum50 for edited videos, SumMe for short raw videos, and ADL for long raw videos.The framework is compared with popular edited- and raw-video summarization approaches.Source: §IV · p. 7
- The experiments compare the framework with various popular approaches to edited- and raw-video summarization.The supplied passage identifies the comparison scope but does not report the outcomes.Source: §IV · p. 7
- The reported dataset coverage includes both edited videos and two raw-video settings: short and long.The datasets named are TVsum50, SumMe, and ADL, respectively.Source: §IV · p. 7
A. Implementation details
Implementation uses a mixed training set of edited and raw videos, deep frame features, and separately learned property-weights for the two video types.
A. Implementation details
- The training set contains 15 edited videos from TVsum50 and 15 raw videos from SumMe, with more than 4,000 annotated shots.Ground-truth summaries are constructed from high-score shots.Source: A. Implementation details · p. 7
- Edited videos generally receive larger be values and raw videos larger br values, emphasizing within-type videos during weight learning.The passage says the coefficients’ sorted distributions reflect this emphasis.Source: A. Implementation details · p. 7
- VGGnet-16 fc6 features represent frames as 4,096-dimensional vectors, and each shot feature averages its frame vectors.The network is pretrained; the paper uses the fc6 layer for frame features.Source: A. Implementation details · p. 7
- The property-weight learning procedure runs 50 times on randomly selected sets of 30 training videos, and reports mean weights for edited and raw videos.Each selection contains 15 edited and 15 raw videos.Source: A. Implementation details · p. 7
B. Results on edited videos
On TVsum50, the framework outperforms three comparison approaches and performs comparably to a method using video titles. Ablations also report lower F-measures when property models or the mixing-coefficient are omitted.
B. Results on edited videos
- Fig. 5 identifies separate property-weight vectors for edited- and raw-video summarization.The caption names the vectors we and wr, respectively.Source: Figure 5 · p. 8
- The framework outperforms VSUMM, LiveLight, and KVS on TVsum50, and achieves comparative performance to title-informed CA without video-title information.Table I reports the comparison; the supplied passages do not provide its numerical values.Source: Table I · p. 8, B. Results on edited videos · p. 8
- Table I reports results for various approaches on TVsum50, the dataset used for edited-video evaluation.The evaluation uses F-measure against ground-truth summaries, matching shots only when they are exactly the same.Source: B. Results on edited videos · p. 7, Table I · p. 8
- Omitting any of the four property models reduces F-measure, with reductions positively correlated with their weights.The resulting summaries also contain more redundant, non-informative, or non-representative shots than the complete framework’s summary.Source: B. Results on edited videos · pp. 8–9, Figure 6 · p. 9
- Training with edited videos alone or with all mixing-coefficients set to 1 yields lower F-measures than the complete framework.The paper attributes the greater decline for train−be to structure mess from the rough mixture, whose interference outweighs its contribution.Source: B. Results on edited videos · p. 9
C. Results on short raw videos
The framework performs best among the compared approaches on SumMe and outperforms three baselines on ADL. Its long-video evaluation counts activities that viewers can recognize in summaries, with repeated activities in distinct video parts counted separately.
C. Results on short raw videos
- On SumMe, quality is measured by mean pairwise F-measure against human summaries, using frame-level overlap because segmentation patterns can differ.The dataset has 25 raw videos lasting 1–6 minutes, each with about 15–18 human-made summaries.Source: C. Results on short raw videos · p. 9
- On SumMe, the framework achieves the best performance among the compared baselines and state-of-the-art approaches.Comparisons include random sampling, uniform sampling, clustering, CSUV, and LSMO.Source: C. Results on short raw videos · pp. 9–10, Table III · p. 10
- Removing any property model lowers F-measure, while combined training helps when the mixing-coefficient is applied.The reported comparisons include modified framework versions and the mixed training set.Source: C. Results on short raw videos · p. 10, Table IV · p. 10
D. Results on long raw videos
- ADL contains 20 daily-life videos, most about 30 minutes long, with activities annotated over time and shots segmented every three seconds.The dataset contains 32 action types, with about 18 in each video.Source: D. Results on long raw videos · p. 11
- ADL evaluation uses the percentage of original-video activities captured in the summary because long videos yield little agreement on a unified human-made summary.Activities appearing in distinct parts of a video are counted repeatedly; viewer recognition determines whether a labeled activity is captured.Source: D. Results on long raw videos · p. 11
- On ADL, the framework outperforms three baselines, and its summaries capture most original activities while reducing redundancy.Summaries are shorter than 10% of the original video duration.Source: D. Results on long raw videos · p. 11, Table V · p. 11
E. Discussion
Results across edited, short raw, and long raw videos support the framework’s effectiveness. The discussion attributes this to the four property models, video-specific property weights, and mixing-coefficients that reduce interference in combined training.
V. CONCLUSION
- The framework’s effectiveness is reported across edited videos, short raw videos, and long raw videos.The conclusion describes its evaluation on three datasets spanning those video types.Source: E. Discussion · p. 11, §V · p. 12
E. Discussion
- Experiments indicate that all four property models are necessary for edited-video and raw-video summarization.The models are importance, representativeness, diversity, and storyness.Source: E. Discussion · p. 11
- Results support learning separate property-weights for edited-video and raw-video summarization rather than using a common weight.Source: E. Discussion · p. 11
- The combined training set is helpful only with mixing-coefficients, which reduce interference from inter-class training videos.Source: E. Discussion · p. 11
- The framework can extend to more video classes by learning one property-weight per class and adding mixing-coefficients.The paper says the property models remain applicable and the additional computation is learning more weights.Source: E. Discussion · p. 11