PaperlayerView exact revision on arXiv

Source-linked AI summary

Less Is More: Picking Informative Frames for Video Captioning

Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang

arXiv:1803.01457v1cs.CV

TL;DR

Video captioning systems often process regularly sampled frames, leaving redundancy, noise sensitivity, and computation costs unresolved. The paper introduces reinforcement-learning-based PickNet to select frames using visual diversity and textual discrepancy within an Encoder-Decoder framework. It reports competitive captioning with only 6–8 frames, while noting that comparison-time estimates are rough.

  • Problem

    Existing attention-based video-captioning methods commonly model regularly sampled frames, which can retain redundant information, content noise, and unnecessary computation.

  • Method

    PickNet sequentially selects informative frames with reinforcement-learning rewards that maximize visual diversity and minimize textual discrepancy, updating the Encoder-Decoder representation after selection.

  • Results

    6–8 frames achieve competitive or comparable video-captioning performance while the architecture reduces convolutional operations across popular benchmarks.

  • Takeaways & Limitations

    A compact selected-frame subset can represent video information for captioning without performance degradation, and PickNet is designed as a plug-and-play complementary component.

  • Takeaways & Limitations

    Compared approaches have rough estimated processing times, so the reported speedup comparisons are approximate.

Abstract

from arXiv · show

In video captioning task, the best practice has been achieved by attention-based models which associate salient visual components with sentences in the video. However, existing study follows a common procedure which includes a frame-level appearance modeling and motion modeling on equal interval frame sampling, which may bring about redundant visual information, sensitivity to content noise and unnecessary computation cost. We propose a plug-and-play PickNet to perform informative frame picking in video captioning. Based on a standard Encoder-Decoder framework, we develop a reinforcement-learning-based procedure to train the network sequentially, where the reward of each frame picking action is designed by maximizing visual diversity and minimizing textual discrepancy. If the candidate is rewarded, it will be selected and the corresponding latent representation of Encoder-Decoder will be updated for future trials. This procedure goes on until the end of the video sequence. Consequently, a compact frame subset can be selected to represent the visual information and perform video captioning without performance degradation. Experiment results shows that our model can use 6-8 frames to achieve competitive performance across popular benchmarks.

1. Introduction

Existing video-captioning methods process regularly sampled frames, which can preserve redundancy, amplify noise, and increase computation. PickNet instead selects informative frames sequentially using visual and textual rewards, aiming to retain captioning performance with a compact subset.

  • Problem: Equal-interval sampling can select duplicated frames, incur substantial feature-extraction costs, and include noisy content such as blur, occlusion, and zoom-out.Existing attention models also process all selected frames, leaving temporal redundancy unresolved.
  • Approach: PickNet is a plug-and-play reinforcement-learning frame selector built on a standard Encoder-Decoder captioning framework.It sequentially evaluates candidates and updates the latent representation when a candidate is selected.
  • Approach: Each frame-picking reward combines visual diversity with textual agreement between the generated caption and the ground truth.The visual objective compares a candidate with already selected frames, while the textual objective minimizes caption discrepancy.
  • Implications: The framework is presented as complementary to other state-of-the-art video-captioning solutions and potentially adaptable to other video-analysis objectives.The paper identifies flexibility as a design merit beyond the Encoder-Decoder framework.

2. Related Works

Visual captioning translates visual content into language and commonly uses CNN–RNN Encoder-Decoder systems with attention. Related video-analysis work addresses temporal selection or summarization, but PickNet selects frames using both semantic and visual information.

  • Visual captioning: Visual captioning translates image or video content into natural language, with modern systems commonly using CNN and RNN Encoder-Decoder architectures.Earlier approaches combined attributes or recognized objects with language models and sentence templates.
  • Visual captioning: Attention-based captioning weights input features spatially for images and temporally for videos.Video attention can model local and global temporal structure while associating visual content with generated words.
  • Video summarization: Video summarization methods select important visual concepts using image searching, canonical visual concepts, or sparse learning.These approaches frame selection as a summarization problem rather than directly as caption generation.
  • Frame selection: Some action-detection systems use frame selection to reduce whole-video processing, but global-position prediction requires knowing the video length.PickNet differs by selecting frames from semantic and visual information and is described as applicable to video streams.

3. Method

The method combines an Encoder-Decoder captioning model with a reinforcement-learning-based PickNet that sequentially selects informative frames. Its rewards balance language quality, visual diversity, and limits on the number of selected frames, while staged training integrates both components.

  • Framework: The framework combines an Encoder-Decoder sentence generator with PickNet for informative frame selection.The Encoder-Decoder produces a fixed-size video representation for caption generation, while PickNet selects the frames supplied to the encoder.
  • Frame selection: PickNet makes sequential pick-or-drop decisions from the current observation and history without global video information.It uses a lightweight two-layer network and a flattened difference grayscale image derived from the current frame and the last picked frame.
  • Frame selection: Selected frames are encoded into the video representation, while dropped frames can allow the convolutional network to stop early and save computation.The first frame is forced to be selected so the encoder processes at least one frame; alternative PickNet inputs such as optical-flow differences are also possible.
  • Rewards: The reward combines CIDEr-based language quality with visual diversity and penalties for selecting too many or too few frames.The visual diversity reward uses the standard deviation of selected visual features; the minimum picked number is 3, while the maximum begins at 1/3 of the total frames and shrinks toward τ.
  • Training: Training proceeds through supervision, reinforcement, and adaptation stages using back-propagation for the Encoder-Decoder and REINFORCE for PickNet.The reinforcement stage fixes the Encoder-Decoder as an environment, and the adaptation stage jointly integrates the pretrained components despite nondifferentiable pick actions.

4. Experimental Setup

The experiments use MSVD and MSR-VTT, evaluate caption quality with standard language metrics, and process sampled appearance features with staged training.

  • Datasets: MSVD contains 1,970 YouTube video clips, while MSR-VTT contains 10,000 video clips with 200,000 video-caption pairs.MSVD clips average about 9 seconds and mainly depict single actions.
  • Evaluation: The evaluation uses BLEU, ROUGEL, METEOR, and CIDEr, with METEOR and CIDEr serving as the main comparison metrics.CIDEr is also used as the language reward.
  • Visual features: Each video is represented by 30 equally spaced 224×224 frames encoded with ResNet152 into 2,048-dimensional appearance vectors.The model omits motion features because their extraction is time-consuming.
  • Text preprocessing: Words occurring fewer than three times are removed, yielding vocabularies of 5,491 words for MSVD and 13,065 for MSR-VTT.Words are represented with one-hot vectors.
  • Training: Training uses learning rates of 3 × 10−4, 3 × 10−4, and 1 × 10−4 across three stages, with batch sizes 128 for MSVD and 256 for MSR-VTT.Each stage runs for up to 100 epochs, and the minimum maximum-pick value is 7.

5. Results and Discussion

PickNet selects informative frames for captioning on MSVD and MSR-VTT, achieving strong captioning results while substantially reducing the number of processed frames. Its learned picks are usually few and tend to occur earlier in videos.

  • Comparison with the state-of-the-arts: 76.0−65.8 ≈15.5% is the reported CIDEr margin over the most recent state-of-the-art method on MSVD.The paper reports that PickNet achieves state-of-the-art performance on MSVD.
  • Comparison with the state-of-the-arts: The MSVD comparison includes six state-of-the-art methods, random selection, k-means selection, an all-frame baseline, and reward ablations.The paper reports that PickNet improves plain techniques and achieves state-of-the-art performance on MSVD.
  • Comparison with the state-of-the-arts: PickNet is reported to be 4∼33 times faster than other methods in the authors’ running-time estimate.The estimate is based on visual-feature-extractor complexity and the number of processed frames.
  • Comparison with the state-of-the-arts: The MSR-VTT comparison includes ruc-uva, Aalto, DenseCap, and MS-RNN, while PickNet is trained end-to-end without attribute information.The supplied passage introduces the comparison but does not include the corresponding numerical results.
  • Analysis of learned picks: In most test videos, PickNet selects fewer than 10 frames, averaging about 6 on MSVD and 8 on MSR-VTT.The paper states that roughly 10/30 frames may be sufficient in most cases, reducing computation cost.
  • Analysis of learned picks: The probability of selecting a frame decreases over time, following a power-law pattern in the reported position distributions.The paper relates this pattern to single-shot videos whose earlier frames can represent the whole video.
  • Online video captioning: In an online setting, frames are sampled at 1fps and fed sequentially to PickNet for immediate caption generation as visual information changes.This setting targets unbounded streaming video rather than offline clips.

6. Conclusion

The conclusion presents PickNet as a plug-and-play reinforcement-learning frame selector that maintains captioning performance while using only a small frame subset. The authors emphasize its efficiency and potential flexibility for other video tasks.

  • Conclusion: PickNet selects informative frames for video captioning and achieves promising effectiveness, efficiency, and flexibility on popular benchmarks.The conclusion characterizes it as a plug-and-play reinforcement-learning-based architecture.
  • Conclusion: Using only 6∼8 frames per video clip, PickNet largely cuts convolution operations compared with methods that usually require more than 40 frames.The authors connect this property to applicability in real-world video processing.
  • Conclusion: The architecture could potentially be employed for video classification and action detection, which the authors leave for future work.The passage presents these applications as potential extensions rather than evaluated results.

7. Details on Time Estimation

The time comparison estimates processing cost from feature-extractor complexity and processed-frame counts under a fixed video input. The authors caution that estimates for competing methods are rough and may understate PickNet’s actual speed advantage.

  • Estimation procedure: The running-time analysis treats PickNet (V+L) as the baseline and accounts for appearance and motion features when estimating visual cost.Motion-feature extraction is treated as doubling total computation time.
  • Dataset-specific estimates: The MSVD and MSR-VTT time tables use average picked-frame counts of about 6 and 8, respectively.Table 3 specifies optical flow and ResNet50 for BA, while Table 4 specifies improved dense trajectory and ResNet50 for DenseCap.
  • Estimation procedure: The comparison fixes each input video at 10 seconds and 36fps, or 360 total frames.Tables 3 and 4 report the resulting estimates for MSVD and MSR-VTT.
  • Caveat: The estimated times for competing approaches are rough because their complex pipelines involve additional attention, region extraction, candidate generation, or recurrent encoding.The authors state that actual PickNet speedup will be higher than estimated.

8. More Result Examples

The examples show PickNet selecting compact, semantically relevant frame subsets for offline and online video captioning. Selected frames can represent actions, shot changes, and evolving content while supporting more appropriate descriptions.

  • Offline video captioning: PickNet selects only a minor portion of frames when videos contain largely duplicated content.In some examples, only three or four frames are picked, concentrated in the video's front portion.
  • Offline video captioning: Selected frames are concise and closely related to generated descriptions, while redundant visual or semantic frames are ignored.The examples include frames representing holding, aiming, shooting, guitar playing, and conversational shot changes.
  • Offline video captioning: Adjacent frames can be selected together when they are needed to represent an action such as seasoning, chopping, or talking.The examples include the 6th and 7th frames and the 15th and 16th frames.
  • Offline video captioning: The generated descriptions may be more accurate than the ground-truth descriptions in some examples.One generated caption identifies a woman instead of using “someone,” while another correctly describes the video content rather than calling it a movie clip.
  • Online video captioning: For online captioning, frames are sampled at 1fps and fed sequentially to PickNet, with descriptions updated as informative frames are selected.The encoder receives features from picked frames and the decoder generates descriptions from the encoded video stream.
Loading 1803.01457v1…