PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Youngjae Yu, Hyungjin Ko, Jongwook Choi, Gunhee Kim

arXiv:1610.02947v3cs.CV

TL;DR

Video-to-language models need useful semantic information for captioning, retrieval, and question answering. The paper proposes an end-to-end concept word detector trained without external knowledge and integrates detected concepts through semantic attention. It achieves the best accuracies on three of four LSMDC 2016 tasks and comparable performance on the fourth.

  • Problem

    Video-to-language tasks require semantic information for captioning, retrieval, and question answering across visual content.

  • Method

    An end-to-end detector learns concept words from caption training data without external knowledge and uses semantic attention to fuse them with language encoding and decoding.

  • Results

    The approach achieves the best accuracies in fill-in-the-blank, multiple-choice test, and movie retrieval, with comparable performance in movie description.

  • Takeaways & Limitations

    Detected concept words can serve as semantic priors for multiple video-to-language models and tasks.

Abstract

from arXiv · show

We propose a high-level concept word detector that can be integrated with any video-to-language models. It takes a video as input and generates a list of concept words as useful semantic priors for language generation models. The proposed word detector has two important properties. First, it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in an end-to-end manner jointly with any video-to-language models. To maximize the values of detected words, we also develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in four tasks of LSMDC 2016. Our approach achieves the best accuracies in three of them, including fill-in-the-blank, multiple-choice test, and movie retrieval. We also attain comparable performance for the other task, movie description.

1. Introduction

The paper introduces an end-to-end concept word detector that supplies semantic priors to video-to-language models without external knowledge sources. Semantic attention integrates detected concepts with language encoding and decoding, and evaluation across four LSMDC 2016 tasks shows strong results.

  • Motivation and approach: The detector generates high-level concept words from videos and associated sentences for captioning, retrieval, and question answering.The words serve as semantic priors for downstream video-to-language models.
  • Motivation and approach: The detector learns word–video-region correlations from caption training data without external knowledge sources and trains jointly with video-to-language models.Its loss can be added as an auxiliary term to the overall model cost.
  • Motivation and approach: Semantic attention selectively focuses on detected concepts and fuses them with input-word encoding and output-word decoding.This provides concept information to encoder hidden states and word prediction.
  • Evaluation: The approach achieves the best accuracies in fill-in-the-blank, multiple-choice test, and movie retrieval, with comparable performance in movie description.These are the four LSMDC 2016 tasks evaluated in the paper.
  • Relation to prior work: Unlike prior video-captioning approaches using external classifiers or separate training stages, this work learns concept detection and language generation end-to-end from captioning data.It also extends semantic attention beyond video captioning to retrieval and question answering.

2. Detection of Concept Words from Videos

The concept detector processes video frames with convolutional features and tracing LSTMs that maintain spatially consistent attention across time. It converts these traces into concept-word confidence scores, trains against caption-derived labels, and selects the highest-scoring words at inference.

  • Video preprocessing: Video frames are sampled uniformly, capped at Nmax = 40, and encoded into 4 × 4 × 500 spatial feature maps with a pretrained ResNet CNN.The spatial reduction accelerates detector training while preserving the stated feature representation.
  • Concept prediction: Candidate concepts are selected from frequent caption nouns, verbs, and adjectives; inference returns the top K = 10 words from V = 2,000 candidates.Training labels mark whether each candidate occurs in the ground-truth caption, and the concept loss is an auxiliary end-to-end objective.
  • Concept traces: The detector uses L tracing LSTMs, each tracking one spatially consistent meaning across frames, with L = 4 × 4 = 16 for the visual feature grid.The traces can correspond to moving objects or actions and are fused to discover concept words.
  • Concept traces: Soft spatial attention is maintained over each frame’s 4 × 4 grid and updated through time so tracked meanings remain temporally consistent rather than frame-specific.The attention weights indicate which spatial locations relate to the concept being tracked.
  • Concept prediction: The detector predicts a vocabulary-sized confidence vector by concatenating the final hidden states of all tracing LSTMs, applying a linear transform, and using sigmoid activation.The sigmoid formulation supports independent multi-label concept predictions.

3. Video-to-Language Models

The models use a shared concept word detector and semantic attention mechanism across several LSMDC video-to-language tasks, while task-specific base models handle each objective. Semantic attention incorporates detected concepts into encoding and decoding, and training jointly optimizes language generation with concept discovery.

  • Shared components: Task-specific base models share the concept word detector and semantic attention mechanism across the four LSMDC tasks.The tasks include movie description, fill-in-the-blank, multiple-choice testing, and movie retrieval.
  • Movie description: The captioning model combines video encoding and caption decoding LSTMs with two semantic attention modules.The video encoder produces hidden states, and the decoder is initialized from the final encoder state.
  • Semantic attention: Semantic attention is applied in input and output pathways to focus dynamically on detected concept words during language modeling.Input attention guides the decoder’s focus at each step, while output attention weights concepts when generating an output word.
  • Training: Training minimizes word-level negative log-likelihood with attention regularization and a concept-discovery loss.The regularized objective uses hyperparameters λ1 and λ2; models for other tasks fine-tune the detector transferred from the description model.
  • Fill-in-the-blank: The fill-in-the-blank model uses a bidirectional LSTM and applies semantic attention to both its input and output representations.The blank is represented with a special token, and prediction is added at the blank position.

4. Experiments

Experiments evaluate the approach on all four LSMDC 2016 video-to-language tasks, with strong benchmark rankings and qualitative evidence that detected concepts support task predictions while missing rare or subtle actions can cause errors.

  • Evaluation Setting: The evaluation covers movie description, fill-in-the-blank, multiple-choice, and movie retrieval under the LSMDC 2016 protocols.The dataset contains 118,114 sentences and 118,081 clips sampled from 202 movies.
  • Movie Description: The approach ranks first in four movie-description metrics and remains comparable to state-of-the-art methods.It ranks (5, 4, 1, 1)-th in BLEU metrics and (2, 1, 1)-th in other language metrics.
  • Movie Fill-in-the-Blank: The fill-in-the-blank model outperforms all participants and baseline LSTM/BLSTM variants, including the base model without concept detection and semantic attention.The evaluated ensemble averages output word probabilities from three independently trained identical models.
  • Movie Multiple-Choice Test: The multiple-choice system ranks first, while a six-model ensemble raises accuracy from 63.8% to 67.0%.The passage attributes additional benefit to the concept detector and semantic attention.
  • Movie Retrieval: The retrieval ensemble achieves the best retrieval performance with significant margins from baselines using averaged similarity matrices.The final matrix combines six retrieval-model matrices with one matrix from the multiple-choice version of CT-SAN.
  • Qualitative Results: Qualitative examples show concept words matching clip content and helping predictions, but rare words and subtle movements remain difficult.Concepts capture relations such as swimming, water, and pool, whereas twitch and cocks are missed in a near-miss retrieval example.

5. Conclusion

The approach detects concept words end-to-end as semantic priors and uses semantic attention to exploit them across multiple video-to-language models. It was evaluated on four LSMDC 2016 tasks and won three.

  • The method detects concept words end-to-end for use as semantic priors in multiple video-to-language models.
  • Semantic attention exploits detected concept words within video-to-language models.
  • The approach won fill-in-the-blank, multiple-choice test, and movie retrieval among the four LSMDC 2016 tasks.It attained comparable performance for movie description.
  • Multiple-choice test: The multiple-choice model scores five candidate sentences for each video and selects the sentence with the highest score.
  • Multiple-choice test: The model uses video encoding to initialize sentence encoding and applies input semantic attention to candidate sentence representations.

A.2. A Model for Retrieval

The retrieval model scores query-text and video pairs using joint multimodal representations built from their encoded sequences. It ranks videos by these scores and improves retrieval through score-matrix ensembling.

  • The retrieval model computes query-text/video scores by learning a joint representation with Compact Bilinear Pooling.The pooling layer captures cross-modal interactions beyond simple concatenation.
  • Video and query text are encoded with LSTMs, using the final hidden state of each sequence as its representation.
  • The model transforms the multimodal representation through a projection, maxout layer, and final score multiplication.The maxout layer yields non-sparse activations while mitigating overfitting.
  • At test time, videos are ranked for each query using similarity scores, with the final model averaging multiple score matrices.The retrieval objective encourages aligned video-sentence pairs to score higher than misaligned pairs by a margin, with ∆= 3 in the experiments.
  • Figure 5 presents architectures for fill-in-the-blank, multi-choice, and movie retrieval, all using concept word detection and semantic attention.

B.1. Implementation Details

The models use standardized optimization and regularization choices, task-specific dataset splits, and ensembles for several evaluations. Retrieval combines multiple similarity matrices, while CT-SAN varies the number of detected words and attention placement in supplementary comparisons.

  • Optimization: All models use Adam optimization, Xavier initialization, two-layer LSTM or BLSTM networks, layer normalization, and dropout at rate 0.2.The initial learning rate ranges from 10^-4 to 10^-5.
  • Optimization: The concept word detector is initialized from a pre-trained movie-description model, after which the whole model is trained end-to-end on each provided training set.
  • Movie Description: The movie-description training split contains 101079 training, 9578 validation, 10053 test, and 7409 blind-test video-sentence pairs.
  • Fill-in-the-blank: Fill-in-the-blank uses 296961 training, 98483 validation, and 30350 test examples, with answer probabilities averaged across three model copies.
  • Additional comparisons: Tables vary the number of detected concept words and compare semantic attention applied only to inputs versus only to outputs.
  • Movie Retrieval: Retrieval averages similarity matrices from six retrieval models and one multiple-choice model over 1,000 sentences and 1,000 movie clips.

C. More Experimental Results

The supplementary section provides additional experimental results intended to support the validity of the concept word detector and semantic attention models.

  • The additional experiments are presented to support the validity of the proposed concept word detector and semantic attention models.

C.1. On the Quality of Concept Words

The ablations show that concept-word quality and end-to-end spatially attended detection are important for improving video-to-language performance.

  • Detector design: CT-SAN outperforms Base-SAN, NN-SAN, and no-ATT-SAN across all four tasks.The largest CT-SAN versus no-ATT-SAN gains occur in fill-in-the-blank and multiple-choice tasks.
  • Detector design: Spatial attention improves concept-word detection, because no-ATT-SAN is weaker than the full CT-SAN model.The no-ATT-SAN variant replaces spatial attention with a single two-layer LSTM over average-pooled visual features.
  • Concept-word quality: Random concept words perform much worse than Base-SAN, showing that semantic attention alone cannot compensate for poor concept words.The rand-SAN baseline uses 10 randomly sampled words from V candidates.
  • Concept-word quality: Nearest-neighbor concept words are less helpful for training semantic attention than concept words detected by the proposed model.NN-SAN copies concepts from the closest training video based on temporally averaged ResNet features.
  • Training strategy: Decoupling concept-word detection from language-model training produces worse performance than end-to-end CT-SAN training.This comparison supports joint learning of the detector and video-to-language model.

C.2. Ablation Study

The ablation studies examine semantic attention and the number of detected concept words, finding limited sensitivity to word count but practical costs from larger lists.

  • Semantic attention: The additional semantic-attention ablation reports results for movie description and fill-in-the-blank in Table 6.The experiment specifically targets the contribution of semantic attention.
  • Number of concept words: Changing K among 5, 10, and 20 detected concept words causes only marginal performance differences.The comparison uses CT-SAN on the evaluated tasks.
  • Number of concept words: Larger concept-word lists increase training time and make overfitting more likely.The passage identifies these as practical disadvantages of increasing K.

D. More Examples and Qualitative Results

Qualitative visualizations and examples illustrate how the detector attends to high-level video concepts and how detected words accompany outputs across the evaluated tasks.

  • Spatial attention: Spatial attention maps roughly capture high-level concepts in videos, such as a blue car moving from left to right.Figures 6–7 visualize these attentions in the movie description model.
  • Movie description: Movie-description examples compare CT-SAN and baselines through generated descriptions and their detected concept words.The comparisons include no-ATT-SAN, S2VT, and Temporal Attention.
  • Movie description: The movie-description example expands “He sees the face of SOMEONE” into a description involving widened eyes, a glowing sphere, and a surface.The listed concepts include light, glowing, sphere, screen, surface, and watch.
  • Task examples: Additional figures provide positive and near-miss examples for movie descriptions, fill-in-the-blank, and multiple-choice testing.The captions distinguish positive examples from near-miss or wrong examples.
  • Fill-in-the-blank: The fill-in-the-blank example correctly selects “house” for a sentence describing a driveway, garage, and front lawn.The associated concepts include drive, car, front, outside, house, street, and road.
  • Movie retrieval: Movie-retrieval figures show ranked retrieved clips alongside the ground-truth clip, using green for positive examples and red for negative examples.The positive examples display the first five retrieved clips; negative examples also report the ground-truth rank.
Loading 1610.02947v3…