Source-linked AI summary
Video Summarisation by Classification with Deep Reinforcement Learning
Kaiyang Zhou, Tao Xiang, Andrea Cavallaro
TL;DR
Video summarisation needs content-specific selection without the cost and subjectivity of frame-level annotations. The paper uses video-level category labels, a companion classifier, and deep Q-learning with global and dense recognisability rewards. On TVSum and CoSum, it outperforms unsupervised and weakly supervised alternatives while remaining highly competitive with supervised methods.
Problem
Existing unsupervised methods use generic criteria, while supervised methods require expensive and potentially biased frame-level summary annotations.
Source: §1 · p. 1Method
The paper trains DQSN with deep Q-learning, using a classification network to provide global recognisability and dense local relative importance rewards from video-level category labels.
Source: §1 · pp. 2–3, §3.2 · p. 5Results
On TVSum and CoSum, the approach outperforms unsupervised and weakly supervised alternatives and is highly competitive with supervised approaches.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
Video-level category labels can support semantic video summarisation through reinforcement learning without requiring frame-level summary annotations.
Source: §1 · pp. 1–2, §5 · p. 10
Abstract
from arXiv · showhide
Most existing video summarisation methods are based on either supervised or unsupervised learning. In this paper, we propose a reinforcement learning-based weakly supervised method that exploits easy-to-obtain, video-level category labels and encourages summaries to contain category-related information and maintain category recognisability. Specifically, We formulate video summarisation as a sequential decision-making process and train a summarisation network with deep Q-learning (DQSN). A companion classification network is also trained to provide rewards for training the DQSN. With the classification network, we develop a global recognisability reward based on the classification result. Critically, a novel dense ranking-based reward is also proposed in order to cope with the temporally delayed and sparse reward problems for long sequence reinforcement learning. Extensive experiments on two benchmark datasets show that the proposed approach achieves state-of-the-art performance.
1 Introduction
The paper proposes weakly supervised video summarisation using video-level category labels, reinforcement learning, and classification-based rewards to preserve category-related information while removing redundancy.
Motivation
- Video-level category labels provide a scalable alternative to expensive, subjective frame-level summary annotations while supporting content-specific selection criteria.Traditional unsupervised criteria are generic, whereas supervised annotations are costly and biased.Source: §1 · pp. 1–2
Framework
- The framework trains a classification network first, then uses its fixed predictions to reward a deep Q-learning summarisation network.The classifier judges whether generated summaries retain enough information for category recognition.Source: §1 · pp. 2–3, Figure 1 · p. 2
- DQSN sequentially removes frames according to predicted future rewards, explicitly encouraging summaries that preserve category-related information.The approach treats summary generation as sequential decision-making because actions are frame-dependent and classification occurs after the sequence.Source: §1 · pp. 2–3
Rewards
- A global recognisability reward measures whether the classifier can recognise the completed summary, linking summary quality to video-category information.This reward is derived from the companion classification network rather than an additional supervision source.Source: §1 · pp. 2–3
- The local relative importance reward provides dense feedback by checking whether each action changes the recognisability of the partial summary, addressing sparse and delayed rewards.This reward is also obtained from the classification network without additional modules.Source: §1 · p. 3
Contributions
- Across TVSum and CoSum, the approach outperforms unsupervised and weakly supervised alternatives and remains highly competitive with supervised methods.The paper reports this as a central contribution of the proposed framework.Source: §1 · p. 3
2 Related Work
Prior video summarisation methods use generic unsupervised criteria, expensive supervised annotations, or weak labels; this work distinguishes itself through reinforcement learning that models frame interdependencies and delayed rewards.
Existing approaches
- Unsupervised methods commonly cluster frames or optimise generic objectives such as diversity and representativeness, which do not encode video semantics.Supervised methods instead exploit semantics from manually annotated summaries, but those annotations are expensive and potentially biased.Source: §2 · p. 3
Weak supervision
- Weakly supervised methods use video category labels to learn concepts for summarisation without requiring expensive summary annotations.The paper positions its approach within this category while using reinforcement learning for frame selection.Source: §2 · p. 3
Related methods
- Compared with Backprop-Grad, the proposed reinforcement-learning formulation better explores interdependencies among frames and significantly improves performance.Backprop-Grad identifies important clips from back-propagated category gradients, whereas this work uses sequential decision-making.Source: §2 · pp. 3–4
Reinforcement learning
- Unlike the diversity- and representativeness-based reward in DR-DSN [46], this approach uses recognisability and a local dense reward to compensate for delayed global feedback.The experiments report that the approach clearly outperforms the methods in [46].Source: §2 · pp. 3–4
3 Proposed Approach
The proposed approach combines a supervised recurrent classifier with a deep Q-learning summarisation network, using category recognisability and dense rewards to guide sequential frame selection.
3.1 Classification Network
- A bidirectional GRU classifier maps pretrained image features to category probabilities, using average pooling, softmax prediction, and label-smoothed cross-entropy training.The classifier is trained first with video-level category labels and then fixed for summary evaluation.Source: §3.1 · pp. 4–5, §3 · p. 4
3.2 Deep Q-Learning Summarisation Network
- DQSN models summarisation as an MDP where each action keeps or discards the attended frame, and training maximises discounted future rewards with deep Q-learning.The process updates the state after every frame decision and terminates at the end of the sequence or when a remaining-frame threshold is reached.Source: §3.2 · p. 5
- DQSN shares lower layers with the classifier and separately estimates state value and action advantage to predict frame-level action values from past and future information.Separating value and advantage is intended to make Q-value learning more efficient and robust to numerical scale.Source: §3.2 · pp. 5–6
3.2.1 Reward Function
- The hybrid reward combines global recognisability, unsupervised diversity-representativeness, and local relative importance rewards.The global reward scores the completed summary using classifier recognition, while the local reward evaluates rank changes after frame removal and provides prompt feedback.Source: §3.2.1 · p. 6
3.2.2 Optimisation with Experience Replay and Double Q-Learning
- Experience replay and double Q-learning optimise DQSN, while testing selects actions by maximum Q-value and forms duration-constrained summaries from shot-level averaged scores.Double Q-learning uses the current learner to select the next action and the target network to evaluate it; testing does not require category labels.Source: §3.2.2 · p. 7
4 Experiments
Experiments on TVSum and CoSum evaluate DQSN’s weakly supervised training, reward design, efficiency, and qualitative summaries. DQSN outperforms unsupervised and weakly supervised alternatives, remains competitive with supervised methods, and produces faster, category-focused summaries.
4.1 Datasets and Settings
- Evaluation uses F-scores against multiple human summaries on TVSum and CoSum with five-fold cross-validation and shot-based summaries.TVSum has 50 videos across 10 categories, while CoSum has 51 videos across 10 categories.Source: §4.1 · p. 7
4.2 Training Classification Network
- 74% and 88% are the pretrained classifier’s accuracies on TVSum and CoSum, versus 66% and 72% when trained from scratch.Pretraining uses YouTube619, which contains 619 videos, and achieves 89.6% average test accuracy there.Source: §4.2 · p. 8
4.3 Comparison with State-of-the-Art Methods
- 1.0% and 4.3% are DQSN’s improvement margins over DR-DSN on TVSum and CoSum, respectively, while it exceeds Backprop-Grad by 5.9% on both datasets.DQSN outperforms all three supervised methods on TVSum and is slightly below DR-DSN but above the other two on CoSum.Source: §4.3 · pp. 8–9, Table 1 · p. 8
4.4 Ablation Study
- Adding the local reward improves over the global reward, while adding the unsupervised reward further improves F-scores and complements the weakly supervised rewards.The full combination is rg + rl + ru, and the unsupervised reward remains beneficial when combined with rg + rl.Source: §4.4 · p. 9
- Local frame relative importance identifies category-relevant frames whose removal changes the true category’s rank, such as Changing Tire and Surfing frames.These examples support using rank changes as supervision for keeping or removing individual frames.Source: §4.4 · p. 9, Figure 3 · p. 9
4.5 Runtime Efficiency
- 1.43 seconds per video makes DQSN more than 2× faster than Backprop-Grad at 3.21 seconds under the same hardware condition.DQSN requires only a forward pass and consumes less memory because it does not save gradients.Source: §4.5 · p. 10
4.6 Qualitative Results
- DQSN preserves category-related frames and temporal storylines, whereas Backprop-Grad selects repetitive scenes and DR-DSN can select irrelevant frames to increase diversity.The qualitative examples include grooming-dog videos, where DQSN retains details such as the pink-clothed woman.Source: §4.6 · p. 10, Figure 4 · p. 10
5 Conclusion
The paper concludes that DQSN uses video-level category labels and reinforcement learning to guide video summarisation. Its global recognisability and dense rewards capture semantics with scalable annotations and yield strong performance against alternative methods.
5 Conclusion
- DQSN combines reinforcement learning, video-level category labels, a global recognisability reward, and a dense reward for video summarisation.The dense reward is intended to mitigate the credit assignment problem in reinforcement learning.Source: §5 · p. 10
- DQSN captures video semantics using easy-to-obtain video-level labels rather than requiring frame-level annotations.The conclusion reports that the approach outperforms unsupervised and weakly supervised alternatives while remaining highly competitive with supervised approaches.Source: §5 · p. 10