Source-linked AI summary
Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders
Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, Baining Guo
TL;DR
The paper addresses automatic highlight extraction without the paired raw and edited videos required by supervised approaches. It learns recurring highlight sub-events from queried edited videos using a robust recurrent auto-encoder with shrinking exponential loss and bidirectional LSTM cells, and reports promising results approaching supervised-learning quality.
Problem
Automatic highlight extraction is needed for lengthy videos, but existing approaches use restrictive heuristics or supervised training with difficult-to-collect raw and edited video pairs.
Method
The method crawls edited videos for each domain and models common highlight sub-events with a recurrent auto-encoder using shrinking exponential loss and bidirectional LSTM cells.
Results
The proposed unsupervised technique reports promising results approaching supervised-learning quality without requiring pre- and post-edit video pairs.
Takeaways & Limitations
Highlight models can be learned from abundant edited web videos, making the approach scalable without collecting raw counterparts.
Takeaways & Limitations
The approach faces subjective differences in what counts as a highlight, noisy or irrelevant query results, and no unedited counterparts for supervision.
Abstract
from arXiv · showhide
With the growing popularity of short-form video sharing platforms such as \em{Instagram} and \em{Vine}, there has been an increasing need for techniques that automatically extract highlights from video. Whereas prior works have approached this problem with heuristic rules or supervised learning, we present an unsupervised learning approach that takes advantage of the abundance of user-edited videos on social media websites such as YouTube. Based on the idea that the most significant sub-events within a video class are commonly present among edited videos while less interesting ones appear less frequently, we identify the significant sub-events via a robust recurrent auto-encoder trained on a collection of user-edited videos queried for each particular class of interest. The auto-encoder is trained using a proposed shrinking exponential loss function that makes it robust to noise in the web-crawled training data, and is configured with bidirectional long short term memory (LSTM)~\cite{LSTM:97} cells to better model the temporal structure of highlight segments. Different from supervised techniques, our method can infer highlights using only a set of downloaded edited videos, without also needing their pre-edited counterparts which are rarely available online. Extensive experiments indicate the promise of our proposed solution in this challenging unsupervised settin
1. Introduction
The paper frames highlight detection as an unsupervised problem using edited web videos, then models recurring highlight sub-events with a robust recurrent auto-encoder. Its shrinking exponential loss addresses noisy training data, while bidirectional LSTM cells model temporal structure.
- Manual highlight production from lengthy wearable-camera videos is time-consuming, motivating automated highlight-clip generation.
- The method learns highlights from queried edited videos rather than requiring paired raw and edited videos.This uses abundant web data and avoids collecting pre- and post-edit video pairs.
- Highlights are modeled as common sub-events, while infrequent or idiosyncratic snippets are treated as non-highlights.The auto-encoder reconstructs positive highlight instances accurately, with non-highlights represented as outliers.
- A shrinking exponential loss progressively reduces the influence of high-error outliers in noisy crawled training data.
- Bidirectional LSTM cells capture temporal dependencies within highlight sequences such as standing, riding, and falling during surfing.
- The proposed unsupervised technique reports promising results approaching supervised-learning quality without collecting pre- and post-edit video pairs.
2. Related Work
Related work spans highlight detection, video summarization, and novelty detection. The paper distinguishes its highlight-focused objective from comprehensive summarization and motivates an auto-encoder because of its scalability and compatibility with LSTM cells.
- Video Summarization: Video summarization seeks diversity, representativeness, or a comprehensive synopsis, whereas highlight detection focuses on selected domain-specific segments.
- Video Highlight Detection: Traditional highlight detection often targets broadcast sports with domain-specific features that are difficult to generalize to ordinary user videos.
- Video Summarization: Supervised summarization methods can achieve state-of-the-art results but may rely on limited user annotations that bias the learned model.
- Novelty Detection: The recurrent auto-encoder is related to novelty detection and is chosen for scalability, easy parallelization, and seamless integration with LSTM cells.
3. Auto-Encoder-Based Removal of Outliers
The auto-encoder reconstructs its input through hidden representations, enabling compressed representations and novelty detection by comparing reconstruction quality. In this pipeline, it is part of an architecture that models temporal context after feature extraction and pooling.
- Auto-encoder fundamentals: An auto-encoder is trained to reconstruct its own input through a hidden layer and output layer.The encoder maps the input to a hidden representation, which the decoder maps back to an output with the same dimensionality.
- Auto-encoder fundamentals: When the hidden layer is smaller than the input, the auto-encoder can provide a compressed representation of the original signal.
- Video highlight pipeline: The pipeline applies temporal pooling before feeding features into a robust recurrent auto-encoder that captures long-range contextual structure.Videos are segmented into snippets, spatial-temporal features are extracted, and local ordering is preserved during pooling.
- Outlier removal: Novelty detection uses faithful reconstruction of inlier instances and poor reconstruction of outliers to classify unseen inputs.Classification is based on the reconstruction error between an input and its reconstructed output.
4. Our Approach
The approach crawls short, user-edited videos for a domain, converts them into snippet features, and learns highlights with a robust recurrent auto-encoder. Shrinking exponential loss reduces noisy-example influence, while bidirectional LSTM cells model temporal dependencies.
- Pipeline: The system retrieves diverse short-form videos for a domain, segments them into snippets, extracts deep features, and feeds pooled snippet representations to an auto-encoder.Videos are restricted to less than four minutes; snippets are temporally segmented and represented using deep features with temporal mean pooling.
- Pipeline: Highlights are identified through common structure among queried videos, using reconstruction error to distinguish coherent highlight snippets from remaining content.The training set is assumed to contain sufficient highlight coherence for reconstruction-based discrimination.
- Robust loss: The shrinking exponential loss makes training robust to web-crawled negative examples by progressively reducing the influence of high-error outliers.Its exponent λ decreases during training, beginning above 1 to promote convergence and later falling below 1 to reduce outlier impact; λ = 1 recovers squared loss.
- Temporal modeling: The recurrent auto-encoder uses LSTM cells to capture temporal dependencies between successive highlight actions and reconstruct input sequences.LSTM gates learn what to remember, forget, and output, while bidirectional cells model both forward and backward dependencies.
5. Experiments
The experiments evaluate the method on large web-crawled edited-video datasets and compare robust recurrent auto-encoding against alternative unsupervised and supervised approaches. Results show benefits from shrinking loss and temporal modeling, while performance remains competitive with supervised learning despite using only edited videos.
- Datasets: The training set contains more than 6500 short-form videos totaling about 13800 minutes, yielding 442075 snippets across seven domains.The dataset is more than 10× longer than the training set used in and can be expanded further.
- Datasets: Testing uses about 30 manually collected raw videos per domain, annotated by six people, with highlights requiring agreement from at least four annotators.The testing videos total about 700 minutes and do not correspond to training videos.
- Evaluation: The evaluation ranks snippets by reconstruction error and averages top-K hit rates as mean average precision (mAP).Lower reconstruction error corresponds to a higher probability of being a highlight snippet.
- Shrinking Loss: Shrinking exponential loss generally matches or surpasses the standard fixed squared loss, although shrinking too far can ignore inliers.The examined range from λ > 1 to λ < 1 generally works well, while λ = 0.125 may be unfavorable in some cases.
- Unsupervised Comparisons: Adding LSTM cells improves standard auto-encoder performance by more than 10%, from 0.371 to 0.410, indicating the value of temporal context.The robust recurrent auto-encoder also consistently outperforms AE, PCA, and OCSVM across domain categories.
- Supervised Comparison: Against latent ranking SVM on the YouTube dataset, the unsupervised method has a small performance gap on dog, gym, parkour, and surfing despite lacking unedited training counterparts.The comparison is disadvantaged because the supervised method uses edited and unedited video pairs.
- Supervised Comparison: Latent ranking SVM trained on its own data with C3D features performs worse than RRAE on the shared testing categories.The paper suggests limited supervised training data may increase overfitting and reduce generalization to large-scale testing sets.
6. Conclusion
The paper presents a scalable unsupervised highlight extraction method built from an improved auto-encoder. Its shrinking exponential loss reduces sensitivity to noisy web-crawled data, while LSTM recurrence models temporal structure.
- Conclusion: The method combines a shrinking exponential loss with recurrent auto-encoder LSTM cells for scalable unsupervised highlight extraction.The paper identifies generalizing the technique to other video-processing problems as future work.