PaperlayerView exact revision on arXiv

Source-linked AI summary

SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition

Bruno Korbar, Du Tran, Lorenzo Torresani

arXiv:1904.04289v2cs.CV

TL;DR

Long untrimmed videos make dense action recognition costly and vulnerable to irrelevant clips dominating aggregation. SCSampler uses a lightweight saliency model to select clips for classification, achieving improved accuracy and lower computation, including a 7% gain and over 15× cost reduction on Sports1M.

  • Problem

    Dense action recognition is prohibitively expensive on long untrimmed videos, where irrelevant clips can outnumber informative predictions and impair video-level classification.

  • Method

    SCSampler densely scores clips with a lightweight saliency model and applies the costly action classifier only to a selected subset, using efficient video or audio features.

  • Results

    On Sports1M, SCSampler adds 7.0% accuracy over dense ir-CSN-152, reaching 84.0%, while reducing inference from 14 days to 0.65 days.

  • Takeaways & Limitations

    The sampler improves runtime and accuracy across six strong action recognizers and retains strong performance on novel action classes.

  • Takeaways & Limitations

    At inference, the oracle cannot be constructed because it requires the true label and dense evaluation by the costly action classifier.

Abstract

from arXiv · show

While many action recognition datasets consist of collections of brief, trimmed videos each containing a relevant action, videos in the real-world (e.g., on YouTube) exhibit very different properties: they are often several minutes long, where brief relevant clips are often interleaved with segments of extended duration containing little change. Applying densely an action recognition system to every temporal clip within such videos is prohibitively expensive. Furthermore, as we show in our experiments, this results in suboptimal recognition accuracy as informative predictions from relevant clips are outnumbered by meaningless classification outputs over long uninformative sections of the video. In this paper we introduce a lightweight "clip-sampling" model that can efficiently identify the most salient temporal clips within a long video. We demonstrate that the computational cost of action recognition on untrimmed videos can be dramatically reduced by invoking recognition only on these most salient clips. Furthermore, we show that this yields significant gains in recognition accuracy compared to analysis of all clips or randomly/uniformly selected clips. On Sports1M, our clip sampling scheme elevates the accuracy of an already state-of-the-art action classifier by 7% and reduces by more than 15 times its computational cost.

1. Introduction

SCSampler addresses the cost and accuracy problems of dense action recognition on long, untrimmed videos by selecting salient clips before applying the action classifier. Its lightweight sampling models reduce computation while improving recognition, including on Sports1M.

  • Motivation: Long real-world videos make dense clip classification computationally impractical and can contain extended irrelevant segments.Pooling all clips may let uninformative predictions outnumber informative ones.
  • Approach: SCSampler uses an extremely lightweight network to determine each candidate clip’s saliency and select a reduced set for action analysis.Its cost is more than one order of magnitude lower than existing 3D CNN action recognizers.
  • Results: SCSampler improves both runtime and accuracy, achieving a 15× speedup and a 7% accuracy gain over a state-of-the-art classifier on Sports1M.The gains result from restricting the costly action classifier to clips identified as salient.
  • Approach: The sampler includes compressed-video and audio-based variants, with the latter exploiting correlation between audio and visual elements.Combining video- and audio-based samplers can produce further recognition-accuracy gains.
  • Learning objectives: Two learning objectives are evaluated: one optimized for a given clip classifier and another classifier-independent objective usable without retraining for each classifier.The classifier-optimized formulation improves accuracy in some settings, while the classifier-independent formulation offers broader reuse.
  • Results: The sampler’s benefits extend to recognition of novel action classes despite training on specific action classes.

2. Related work

SCSampler differs from video summarization and action localization because it selects fixed-duration clips for video classification rather than producing composites or localizing multiple actions. It also differs from temporal skipping by scoring all segments efficiently before analyzing the most salient ones.

  • Video selection: Video summarization and sport-highlight systems create shorter, compelling composites from informative or exciting snippets.SCSampler instead selects fixed-duration segments for classification.
  • Action localization: Action localization finds temporal boundaries and recognizes potentially different actions, whereas SCSampler targets videos assumed to contain a single action class.Its sampler resembles proposal selection but uses fixed-duration clips for video classification.
  • Efficiency: For a 60-second video, dense MC3-18 classification costs 1097 GFLOPs, while the sampling scheme costs 168 GFLOPs.The comparison contrasts dense clip classification with the lower-cost sampling approach.
  • Temporal selection: Temporal skipping predicts which future frames to consider, whereas SCSampler evaluates all segments with a fast procedure and analyzes only the most salient ones.
  • Video classification: SCSampler uses simple averaging of selected clip scores, consistent with the aggregation scheme used by recent state-of-the-art action classifiers.

3. Technical approach

The technical approach densely scores clips with a lightweight saliency model, selects the top K, and applies the costly action classifier only to those clips. It supports classifier-based and ranking-based training, efficient compressed-video or audio features, and multimodal score combination.

  • Overview: SCSampler extracts a small set of relevant clips by densely scoring every clip with a lightweight saliency model.The approach includes two learning objectives and choices of efficient features.
  • Problem formulation: Dense clip classification aggregates predictions from all clips, but long untrimmed videos make this expensive and can reduce accuracy when the action is absent from many clips.
  • Sampler: The sampler assigns each clip a saliency score, selects the top-K clips, and aggregates the action classifier’s predictions only over that subset.This constrains the costly classifier to K clips and can discard irrelevant or ambiguous clips.
  • Oracle: The oracle selects clips with the highest ground-truth action scores and supplies an accuracy upper bound and pseudo-ground-truth training data.It is impractical at inference because it requires the true label and dense action-classifier evaluation.
  • Action classification objective: The action-classification objective scores clips by their maximum predicted class response, assuming every training-video clip contains the target action.This variant is independent of the given classifier f.
  • Saliency ranking: The saliency-ranking objective trains pairwise clip preferences so clips receiving higher correct-class scores from classifier f are ranked higher.It uses pseudo-ground-truth binary labels for clip pairs and a margin-based ranking loss.
  • Features: Efficiency is supported by compressed-video features that avoid decoding and by low-dimensional audio features that can be processed efficiently.
  • Multimodal combination: Audio and visual saliency scores can be combined through convex-score or ranked-list strategies, including an audio-video union-list method.

4. Experiments

Experiments on Sports1M and Kinetics show that SCSampler improves video-level action recognition while evaluating only a reduced set of clips. The gains persist across classifiers, datasets, and sampler configurations, with substantial efficiency benefits on long videos.

  • Experimental setup: SCSampler is evaluated on Sports1M and Kinetics with six popular 3D CNN action classifiers.Sports1M contains long, untrimmed videos, while Kinetics provides a shorter-video setting.
  • Sports1M: SCSampler selects the top K = 10 clips per Sports1M test video instead of evaluating every clip densely.Runtime includes both sampling and clip-level action classification over the complete test set.
  • Sports1M: 6.0%–9.9% accuracy gains over Empirical are reported across action models, while Empirical performs only marginally better than Random and Uniform.The gains range from 6.0% for R(2+1)D-34 to 9.9% for R(2+1)D-18.
  • Sports1M: 7.0% higher accuracy than Dense raises ir-CSN-152 from 77.0% to 84.0% on Sports1M, while reducing inference runtime from 14 days to 0.65 days.Dense averages predictions over all non-overlapping clips; SCSampler achieves the higher accuracy using selected clips.
  • Sampling behavior: SCSampler’s selected clip locations resemble Oracle’s on Sports1M, with many samples near the beginning and especially the end of videos.On Kinetics, the Oracle and SCSampler location distributions are much more uniform.
  • Kinetics: SCSampler improves accuracy over Random, Uniform, Empirical, and Dense for all architectures on Kinetics, despite its videos averaging about 10 seconds.The ir-CSN-152 configuration reaches 80.2% accuracy, reported as the best result on that benchmark.
  • Sampler configurations: AV-union-list is the best audio–video combination tested, using K′ = 8 visual-sampler clips and K − K′ audio-sampler clips when K = 10.The combined strategy slightly improves over the visual sampler alone, despite the visual sampler being the stronger selector.

5. Discussion

SCSampler uses a lightweight model to select a small subset of clips, improving both the accuracy and speed of clip-based action classifiers. Its benefits extend across multiple recognizers and novel action classes, while future work targets redundancy-aware sampling and richer temporal aggregation.

  • SCSampler selects a small subset of clips to improve both the accuracy and speed of clip-based action classifiers.
  • Its clip-sampler produces large accuracy gains and substantial speedups across six strong action recognizers.
  • The sampler retains strong performance when recognizing novel action classes, despite training on specific classes.
  • Future work will address redundant sample-set selection and extensions to models using more sophisticated temporal aggregation.

A.1. Architecture details

The evaluated action classifiers include mixed-convolution and decomposed spatiotemporal residual architectures. MC3 uses 3D convolutions early and 2D convolutions later, while R(2+1)D factorizes each 3D convolution into spatial and temporal operations.

  • MC3 is a mixed-convolution model whose first x −1 convolutional groups use 3D convolutions and later groups use 2D convolutions.
  • R(2+1)D decomposes each 3D convolution into a spatial 2D convolution followed by a temporal 1D convolution.

A.2. Training procedure

The action classifiers are trained with dataset-specific clip sampling, resizing, cropping, batch, and learning-rate schedules. Sports1M uses 100 training epochs, while Kinetics uses 45 epochs with temporal jittering.

  • Sports1M clips use 16 adjacent frames and a shared 224 × 224 random crop after resizing frames to 342 × 256.
  • Sports1M models use batches of 8 clips per GPU and train for 100 epochs, including 15 warm-up epochs.
  • Kinetics mini-batches sample five 16-frame clips with temporal jittering, using the same 342 × 256 resizing and 224 × 224 cropping scheme.
  • Kinetics models train for 45 epochs with 10 warm-up epochs and a learning rate of 0.01 reduced every 10 epochs.

B.1. Visual-based sampler

Visual samplers are pretrained on ILSVRC and trained on Sports1M and Kinetics with a learning rate of 0.001. Pretrained layers use learning rates 100 times smaller, and ShuffleNet0.5 uses a specified grouped-convolution configuration.

  • All visual samplers are pretrained on ILSVRC, with a learning rate of 0.001 for both Sports1M and Kinetics.
  • Pretrained layers use learning rates 100× smaller, with reductions applied when accuracy plateaus.
  • ShuffleNet0.5 uses three groups of group convolutions, a configuration selected for its reported accuracy.

B.2. Audio-based sampler

The audio-based sampler uses a pretrained VGG backbone with MEL spectrogram inputs and is trained with specified optimization settings. Additional analyses examine its design and hyperparameters.

  • B.2. Audio-based sampler: The sampler is fine-tuned with SAL-RANK, SAL-CL, or action-classification losses using distinct learning-rate schedules.The schedules vary across Sports1M and Kinetics and across the training loss used.
  • B.2. Audio-based sampler: The paper presents additional analyses of SCSampler design choices and hyperparameter values.

C.1. Varying the audio sampler architecture.

The audio sampler is evaluated with alternative architectural fine-tuning choices on miniSports using MC3-18 and ten sampled clips. Fine-tuning the entire VGG model performs best.

  • C.1. Varying the audio sampler architecture.: Fine-tuning the entire VGG model achieves the best classification accuracy among the evaluated audio-sampler variants.The alternatives train a single fully connected layer on activations from conv4_2, pool4, or fc1.
  • C.1. Varying the audio sampler architecture.: All evaluated audio samplers are trained with the SAL-RANK loss.

C.2. Varying the number of sampled clips (K)

The study varies the number and source of sampled clips for SCSampler on miniSports. With AV-union-list, ten clips perform best, using eight visual-sampler clips and two audio-sampler clips.

  • C.2. Varying the number of sampled clips (K): K = 10 provides the best accuracy for the AV-union-list sampler on miniSports.The corresponding analysis uses MC3-18 as the clip classifier.
  • C.2. Varying the number of sampled clips (K): The Oracle reaches its top result with K = 1 because it selects the clip with the highest correct-label score for each test video.
  • C.2. Varying the number of sampled clips (K): The AV-union-list strategy selects K′ visual top-clips and adds K − K′ different clips from the audio-ranked list.The experiment fixes K = 10.
  • C.2. Varying the number of sampled clips (K): K′ = 8 visual clips and K − K′ = 2 audio clips achieve the best action recognition accuracy.The result suggests complementary signals, with the visual sampler providing a more accurate saliency measure.

D. Comparison to Random/Uniform under the same runtime.

The runtime comparison evaluates SCSampler against Random and Uniform sampling while varying the number of sampled clips. Even after accounting for sampling overhead, SCSampler significantly outperforms these baselines.

  • D. Comparison to Random/Uniform under the same runtime.: Its overhead is roughly equivalent to 3 MC3-18 clip evaluations per video.
  • D. Comparison to Random/Uniform under the same runtime.: SCSampler significantly outperforms Random and Uniform under matched runtime, even after adding their clip evaluations for comparison.
  • D. Comparison to Random/Uniform under the same runtime.: For costlier clip classifiers, SCSampler overhead can fall below one clip evaluation, such as 0.972 for R(2+1)D-50.This makes Random and Uniform less appealing under the same runtime.
  • D. Comparison to Random/Uniform under the same runtime.: Running SCSampler every N clips further reduces computational cost by selecting final top-K clips from a strided subset.The experiment fixes K = 10 and evaluates the best SCSampler configuration with ip-CSN-152.
Loading 1904.04289v2…