PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Video Classification and Captioning

Zuxuan Wu, Ting Yao, Yanwei Fu, Yu-Gang Jiang

arXiv:1609.06782v2cs.CVcs.MM

TL;DR

Understanding video content is central to video applications as video data has expanded rapidly. This chapter reviews deep-learning research in video classification and captioning, along with benchmarks and challenges, and concludes that the tasks rely on spatial and temporal information. It describes classification as learning robust representations from raw video and captioning as jointly modeling visual understanding and language.

  • Problem

    Video understanding is a fundamental issue for applications such as retrieval and surveillance, while video captioning must capture objects, activities, and their relations in natural sentences.

    Source: §1.1 · p. 1, §1.4 · p. 12

  • Method

    The chapter surveys deep-learning techniques for video classification and captioning and reviews popular benchmarks and challenges.

    Source: §1.1 · pp. 1–2

  • Results

    The reviewed classification and captioning approaches rely on modeling spatial and temporal video information; classification derives robust, discriminative representations from raw data, while captioning jointly models visual understanding and language.

    Source: §1.6 · p. 27

  • Takeaways & Limitations

    The chapter presents video classification as a potential fundamental component in video captioning and identifies benchmarks and challenges as critical for evaluating technical progress.

    Source: §1.6 · p. 27

    Generated captions remain far from practical use, with simple sentence forms and limited vocabulary.

    Source: §1.4.3 · p. 17

Abstract

from arXiv · show

Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data. In this paper, we focus on reviewing two lines of research aiming to stimulate the comprehension of videos with deep learning: video classification and video captioning. While video classification concentrates on automatically labeling video clips based on their semantic contents like human actions or complex events, video captioning attempts to generate a complete and natural sentence, enriching the single label as in video classification, to capture the most informative dynamics in videos. In addition, we also provide a review of popular benchmarks and competitions, which are critical for evaluating the technical progress of this vibrant field.

1.1 Introduction

The chapter reviews deep-learning approaches to video classification and captioning against the backdrop of rapidly growing video data and the need to understand its content. It also outlines foundational modules and later coverage of benchmarks and challenges.

1.1 Introduction

  • Video data has spread explosively, making video understanding fundamental to applications such as advertising, retrieval, and surveillance.The chapter notes that advances in image and speech deep learning have motivated video representations that exploit multimodal clues.Source: §1.1 · p. 1
  • Video classification labels clips by semantic content, while captioning generates a natural sentence that captures informative dynamics beyond a single label.Source: §1.1 · pp. 1–2
  • Unlike earlier surveys centered on handcrafted features and conventional machine-learning pipelines, this chapter reviews deep learning for classification and captioning.Source: §1.1 · p. 2
  • The chapter introduces basic deep-learning modules, then reviews classification, captioning, and benchmarks and challenges for evaluating progress.Source: §1.1 · p. 2, §1.2 · pp. 2–3

1.2.1 Convolutional Neural Networks (CNNs)

CNNs developed from early hierarchical image models into deeper architectures designed to improve feature representations and recognition. The section traces key designs and training strategies in that progression.

1.2.1 Convolutional Neural Networks (CNNs)

  • Fukushima’s neocognitron used local neuron connectivity across hierarchically transformed images, preceding CNNs; LeNet-5 later applied end-to-end back-propagation to raw images.LeNet-5 and its variants were limited on more complex vision tasks by training-data scale and computational power.Source: §1.2.1 · p. 3
  • AlexNet combined five convolutional layers with three fully connected layers, using ReLUs for faster training and dropout to alleviate overfitting.Source: §1.2.1 · p. 3
  • VGG16 and VGG19 extended CNN depth to 16–19 layers, while small 3 × 3 filters captured input-image details.Source: §1.2.1 · pp. 3–4
  • GoogLeNet uses multi-scale processing, while Inception architectures increase CNN depth and width while maintaining affordable computational cost.Source: §1.2.1 · pp. 3–4
  • ResNet increased CNN depth to 152 layers through residual layers with skip connections and won first place in the 2015 ImageNet Challenge.ResNet was also extended beyond 1000 layers on CIFAR-10.Source: §1.2.1 · p. 4
  • Across architectures, increasing depth is described as improving feature representations and discriminative power, while methods such as Maxout, DropConnect, and Batch Normalization facilitate training.Source: §1.2.1 · p. 4

1.2.2 Recurrent Neural Networks (RNNs)

RNNs extend feed-forward networks with recurrent state for sequential inputs, but standard RNNs struggle with long-range retention and gradient stability. LSTMs add gated memory, while video classification is shifting from handcrafted features toward deep representations.

1.2.2 Recurrent Neural Networks (RNNs)

  • At each time step, an RNN maps an external input through hidden states to an output, using recurrent connections to preserve a memory of prior inputs.The chapter defines the input and output vectors and the weight matrices and biases used in the block.Source: §1.2.2 · pp. 4–5
  • Standard RNNs struggle to model long-range dependencies and retain past inputs, and training can encounter vanishing or exploding gradients.LSTMs were introduced to address these issues.Source: §1.2.2 · p. 5
  • LSTMs add input, forget, and output gates around a memory cell to control information flow and exploit long-range temporal memory.The forget gate controls retained cell information, the input gate incorporates new information, and the output gate controls information passed to the hidden state.Source: §1.2.2 · pp. 5–6

1.3 Video Classification

  • Video classification has moved from handcrafted spatio-temporal descriptors, bag-of-words or Fisher Vector representations, and classifiers toward deep features learned from raw video.The chapter reviews supervised deep learning and unsupervised feature learning for video classification.Source: §1.3 · pp. 6–7

1.3.1 Supervised Deep Learning for Classification

The review covers image-based video recognition, end-to-end CNNs, recurrent models for longer temporal structure, and visual attention. These approaches range from pooling frame features to combining spatial, motion, and sequential cues.

1.3.1.1 Image-Based Video Classification

  • Off-the-shelf CNN frame features can be averaged into video representations for standard classifiers; VLAD encoding improved on trivial averaging, and FV-VAE was proposed to learn generalized representations.Image-based recognition also used features from different CNN layers with multiple-kernel classifiers.Source: §1.3.1.1 · p. 7

1.3.1.2 End-to-End CNN Architectures

  • 3D CNNs extend image convolutions across video volumes, while one comparison found single-frame CNNs performed similarly to stacked-frame models and mixed-resolution inputs could speed training.Tran et al. applied 3D convolutions to full frames rather than detector-segmented volumes.Source: §1.3.1.2 · pp. 7–8
  • Stacked-frame CNNs were reported below state-of-the-art handcrafted features, while factorized convolutions and two-stream networks addressed training cost and separated spatial from temporal cues.The two-stream approach uses RGB frames for appearance and dense optical flow for motion, then combines stream scores by weighted sum.Source: §1.3.1.2 · p. 8, Figure 1.3 · p. 9, Figure 1.4 · p. 9
  • Two-stream extensions pool features along point trajectories, fuse streams with convolutional layers, combine segment scores, replace flow with motion vectors, or represent motion with dynamic images.A Siamese-CNN approach modeled actions as transformations from an initial condition to a new effect and reported better results than Simonyan and Zisserman [2014] on two benchmarks.Source: §1.3.1.2 · pp. 9–10

1.3.1.3 Modeling Long-Term Temporal Dynamics

  • LSTMs model ordered long-term video dynamics beyond adjacent-frame motion; reviewed approaches combine them with CNN features, while soundtrack spectrogram CNNs add audio cues.One study found CNNs and LSTMs highly complementary; another reported strong results using soundtrack-derived spectrograms alongside visual clues.Source: §1.3.1.3 · pp. 10–11

1.3.1.4 Incorporating Visual Attention

  • Visual attention targets discriminative spatio-temporal volumes, including through soft attention and motion-based attention derived from optical flow.The motivation is that many video frames are computationally expensive to use and may be irrelevant to the class of interest.Source: §1.3.1.4 · p. 11

1.3.2 Unsupervised Video Feature Learning

The review describes the annotation burden in video classification and the potential of unlabeled video for learning representations. It also introduces video captioning as sentence generation that must capture video dynamics, and identifies a common convolutional–recurrent architecture.

1.3.2 Unsupervised Video Feature Learning

  • Unsupervised learning that uses spatial and temporal context is presented as a promising way to learn video structure when scaling labeled datasets is difficult.The passage notes that thousands of categories create major annotation challenges, including at video level and especially for frame-level fine-grained labels.Source: §1.3.2 · p. 11
  • Srivastava et al. [2015] pre-trained an encoder-decoder LSTM on unlabeled YouTube sequences, then fine-tuned it on standard benchmarks for action recognition.The encoder maps an input sequence to a fixed-length representation; decoder LSTMs reconstruct the input or predict future sequences.Source: §1.3.2 · p. 12
  • Video-classification models often adapt image- and speech-domain architectures, while video’s spatial, temporal, and acoustic cues call for tailored modeling, especially of temporal dynamics.The passage also identifies expensive manual labeling as a reason to use unlabeled videos and contextual clues for representation learning.Source: §1.3.2 · p. 12

1.4 Video Captioning

  • Video captioning must produce a natural sentence that captures objects, scenes, and activities and expresses their relationships, not merely recognize video content.The task is described as potentially useful for robotic vision and for helping visually impaired people.Source: §1.4 · p. 12
  • The review groups captioning methods by sentence-modeling strategy and distills a common architecture combining convolutional and recurrent neural networks.Figure 1.6 contrasts outputs from tagging, single-frame captioning, and video captioning, with the latter describing dynamic video content.Source: §1.4 · p. 13, Figure 1.6 · p. 13

1.4.1 Problem Introduction

Video captioning generates a natural sentence from a video, going beyond tags and single-frame descriptions to express semantic and spatio-temporal relationships. The section outlines CNN/RNN-based approaches and more sophisticated frameworks that incorporate knowledge or model sentence–video relationships.

1.4.1 Problem Introduction

  • Video captioning must express semantic content and spatio-temporal relationships in a natural sentence, unlike tags or single-frame captions.The passage illustrates the distinction with a video description of one baby boy biting another baby boy’s finger.Source: §1.4.1 · p. 13
  • Captioning approaches commonly use 2D or 3D CNNs to extract visual representations and LSTMs to generate sentences word by word.Other frameworks integrate high-level semantic attributes or explore relationships between sentence semantics and video content.Source: §1.4.1 · pp. 13–14
  • The review organizes captioning methods by sentence-generation strategy and separately presents a common architecture based on sequence learning.The two method categories are covered in Sections 1.4.2 and 1.4.3, respectively.Source: §1.4.1 · p. 14

1.4.2 Approaches for Video Captioning

The review distinguishes template-based language models from sequence-learning methods for video captioning. Templates constrain sentence structure and align fragments with visual concepts; sequence models learn mappings from video content to more flexible sentences.

1.4.2 Approaches for Video Captioning

  • The review divides captioning into template-based language models and sequence-learning models, which generate sentences through different strategies.Templates prescribe sentence grammar and parts such as subject, verb, and object; sequence models learn a mapping from video content to sentences.Source: §1.4.2 · p. 14

1.4.2.1 Template-based Language Model

  • Template-based approaches align sentence fragments with detected visual concepts and use language constraints to assemble descriptions.Examples include CRF-based descriptions and semantic hierarchies for selecting fragment specificity; Xu et al. [2015b] designed a deep joint video-language embedding model for video sentence generation.Source: §1.4.2.1 · pp. 14–15

1.4.2.2 Sequence Learning

  • Sequence-learning methods map visual content and text in a common space, allowing novel sentences with more flexible syntax than templates.Examples include LSTMs using frame features, frames with optical flow, temporal attention, or semantic attributes; Yao et al. [2015a] use temporal attention and spatio-temporal convolutional features.Source: §1.4.2.2 · p. 15

1.4.3 A Common Architecture for Video Captioning

A common captioning pipeline converts visual features into video-level representations, then generates a sentence word by word with an LSTM. The chapter also notes that generated sentences remain simple and vocabulary-limited.

1.4.3 A Common Architecture for Video Captioning

  • 2D/3D CNN features from frames, optical flow images, or clips are combined by mean pooling or soft attention before an LSTM generates a sentence word by word.The video-level representation conditions the sentence generation.Source: §1.4.3 · pp. 15–16, Figure 1.7 · p. 16
  • The model represents a sentence’s log probability as a sum of sequential word probabilities conditioned on the video and preceding words.Training maximizes the description probability; inference feeds back the most probable word until the end sign is emitted.Source: §1.4.3 · pp. 16–17
  • Generated sentences remain simple and vocabulary-limited, leaving free-form sentence generation and open vocabulary as future issues.Source: §1.4.3 · p. 17

1.5 Benchmarks and Challenges

  • The chapter next reviews popular benchmarks and challenges for video classification and video captioning.Source: §1.5 · p. 17

1.5.1 Classification

This section surveys video-classification datasets and challenges, then summarizes results on UCF-101 and HMDB51. The reviewed benchmarks span varied video sources, label schemes, and task settings.

1.5.1 Classification

  • Large and challenging datasets, including UCF101, HMDB51, and FCVID, and open challenges have stimulated video-classification research.Source: §1.5.1 · p. 17

1.5.1.1 Datasets

  • For video classification, UCF-101 has 13,320 video clips in 101 annotated classes, while ActivityNet has 27,801 video clips annotated into 203 activity classes.ActivityNet totals 849 hours of video.Source: §1.5.1.1 · pp. 19–20
  • Annotation approaches vary: Sports-1M and EventNet use online textual information, whereas FCVID is manually annotated.The chapter notes that Sports-1M labels are not clean, although its authors describe their quality as fairly good.Source: §1.5.1.1 · p. 20

1.5.1.2 Challenges

  • THUMOS evaluates large-scale action recognition, TRECVID MED scores event likelihood, and ActivityNet tests untrimmed classification and temporal detection.TRECVID MED uses AP and inferred AP; ActivityNet detection predicts activity labels and temporal extents.Source: §1.5.1.2 · pp. 20–21

1.5.1.3 Results of Existing Methods

  • Results on UCF-101 and HMDB51 are summarized, with video-classification performance generally measured by AP for a single class and mean AP for multiple classes.The chapter describes the pace of development as fast but supplies no specific comparison in the provided passage.Source: §1.5.1.3 · p. 21

1.5.2 Captioning

This chapter reviews video-captioning datasets, evaluation metrics, and representative methods. It concludes that better video representations and free-form, open-vocabulary descriptions remain challenging research problems.

1.5.2.1 Datasets

  • Captioning datasets differ in annotation structure: MSVD provides roughly 40 English descriptions per video, TACoS-ML pairs 14,105 intervals across 185 long videos with 52,593 sentences, and M-VAD gives each movie clip one sentence.Source: §1.5.2.1 · p. 24

1.5.2.2 Evaluation Metrics

  • BLEU@N measures shared n-grams, METEOR uses unigram precision and recall with synonym and stem matches, and CIDEr weights n-grams with TF-IDF.The text cautions that high-order BLEU matches rarely occur at sentence level, especially when comparing individual sentences.Source: §1.5.2.2 · p. 25

1.5.2.3 Results of Existing Methods

  • Representative captioning methods are evaluated on MSVD, M-VAD, MPII-MD, and TACoS-ML, and the reviewed work is described as recent and fast-developing.Source: §1.5.2.3 · p. 25

1.6 Conclusion

  • Deep learning for classification derives feature representations from raw data, while captioning jointly models visual understanding and language description.The chapter also reviews popular benchmarks and challenges for both tasks.Source: §1.6 · p. 27
  • Learning video representations that capture object interactions and their evolution over time with limited or no supervision remains an open problem.The conclusion also identifies free-form, open-vocabulary video descriptions as a continuing challenge.Source: §1.6 · p. 27

Index

These index segments provide page references and cross-references to topics, methods, and datasets across video classification and captioning, multimedia analysis, similarity search, and computing systems.

Index

  • Video entries cover classification datasets and benchmarks, CNN architectures, captioning approaches, and BLEU@N and CIDEr metrics.The index also lists ActivityNet and Columbia Consumer Videos datasets, plus captioning architectures and conditional random fields.Source: Index · pp. 93–95
  • Similarity-search entries include approximate-neighbor and hash-based methods, query mechanisms, search costs, and encrypted or compressed-domain retrieval.The index also references sketch similarity and similarity-search evaluation measures.Source: Index · pp. 93, 95, 96
  • Computer-audition entries span audio data and backgrounds, segment grouping, sound generation and structure, and applications.Related entries include speech and speaker recognition, blind source separation, and computational auditory scene analysis.Source: Index · p. 94
  • Other indexed topics include situation recognition, emotion and personality recognition, Hawkes processes, and crowdsourced multimedia services.Entries also point to multimodal social-interaction analysis, encrypted multimedia processing, and cloud-based crowdsourcing.Source: Index · pp. 94, 96
  • Cloud-gaming and fog-computing entries cover deployment, streaming, compression, communication, interaction delay, and future directions.The index also names GamingAnywhere, thin-client design, and cloudlet servers.Source: Index · pp. 95–96
  • Deep learning entries point readers to video-classification and captioning topics, including benchmarks, challenges, methods, and datasets.The index lists deep-learning material on pages 3–29, including video classification on pages 8–14 and 19–23 and video captioning on pages 14–19.Source: Index · pp. 97, 99
  • Other cross-references span event and Hawkes processes, similarity searches, cloud and fog computing, and social-interaction analysis.Entries point to topics such as Hawkes processes, similarity-search methods, fog-computing mechanisms, and free-standing conversational groups.Source: Index · pp. 96–99
  • The contents list multimedia-analysis entries for audio processing on pages 95–100, image processing on pages 84–90, and video processing in the encrypted domain.Source: Index · p. 98
  • The segment indexes crowd and infrastructure topics, including CrowdMAC, smartphone-augmented sensing, fog computing, and cloud gaming.Entries also reference fog-system setup and architecture, mobile Internet hotspots, and gaming hardware and interaction delay.Source: Index · pp. 99, 101
  • Similarity-search entries span hashing, indexing, graph-based methods, quantization, and sketch similarity.The listed topics include LSH, KD-trees, K-means, lattices, and hybrid approaches.Source: Index · pp. 100–102
  • The index lists Hawkes-process topics including branching, estimation, simulation, thinning, and generative models.Related entries also cover event intensity, likelihood, parameter estimation, and information diffusion.Source: Index · pp. 100–101
  • Video-classification entries name HMDB51, Hollywood Human Action, Kodak Consumer Videos, and KTH datasets, alongside LSTM and temporal-dynamics topics.The LSTM entries include video classification and long-term temporal dynamics modeling.Source: Index · pp. 100–102
  • The index lists Poisson processes (193–197), Poisson memorylessness (110–111), a Hawkes-process introduction (191–193), marked Hawkes processes (210–211), and a Hawkes power-law kernel function (305–306).Source: Index · pp. 103, 105
  • Computing index entries include mobile offloading in fog computing (263–267), fog computing (280–285), cloud gaming (288, 292), OTT service for cloud gaming (310), power consumption, and Quality of Experience (QoE) metrics in cloud.Source: Index · pp. 103–105
  • The multimedia-analysis index lists multimodal pose estimation (59–66), music signals in audition (37–38), and an entry for object recognition with eye fixations; it also lists head and body pose estimation (57–58).Source: Index · pp. 103–104
  • The index lists neighbors in similarity searches on pages 106–111, near neighbors in similarity searches on page 114, similarity searches on pages 118–119, and sketch similarity on page 122.Source: Index · p. 104
  • Video-captioning entries include sequence learning and a template-based language model.The index points to sequence learning on page 17 and the template-based language model on pages 16–17.Source: Index · pp. 107, 109
  • Video-classification entries cover the TACoS Multi-Level Corpus, Sports-1M, LSTM, TDD, temporal segment networks, and the THUMOS Challenge.The index locates these topics on pages 11, 22–26, and 23, respectively.Source: Index · p. 109
  • The index lists video-classification datasets UCF-101 and THUMOS-2014 at pages 21 and 24, respectively; it also lists Weizmann and YouCook, with YouCook at page 26.Source: Index · pp. 110–111
  • Video classification entries cover its overview, supervised and unsupervised learning, and research topics across pages 8–23.The index also lists video-classification datasets, including TRECVID MED and TV16-VTT.Source: Index · pp. 109–110

Shih-Fu Chang

Shih-Fu Chang is a Columbia Engineering professor whose research spans multimedia retrieval, computer vision, machine learning, and signal processing. His work includes foundational visual-search systems, broad scholarly and professional recognition, and leadership in academic and research organizations.

Shih-Fu Chang

  • Chang’s research spans multimedia information retrieval, computer vision, machine learning, and signal processing, with a goal of extracting rich information from large collections of visual data.His work has included image and video recognition, multimodal analysis, visual content ontology, image authentication, and compact hashing for large-scale indexing.Source: Shih-Fu Chang · p. 113
  • His early visual-search systems VisualSEEk and VideoQ helped set the foundation of content-based visual search.His work has also affected applications including image and video search engines, online crime prevention, mobile product search, AR/VR, and brain-machine interfaces.Source: Shih-Fu Chang · p. 113
  • Chang has authored more than 350 peer-reviewed publications, received multiple technical and teaching awards, and holds more than 30 issued patents.Technologies based on his work have been licensed to seven companies.Source: Shih-Fu Chang · p. 113
  • He has held leadership roles at Columbia and ACM SIGMM and serves as Senior Executive Vice Dean at Columbia Engineering.He is a Fellow of AAAS, IEEE, and ACM.Source: Shih-Fu Chang · pp. 113–114

Related papers

Loading 1609.06782v2…