End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

1. Introduction

• Video captioning [6] [21] [31] [39] and question answering [27] are emerging challenges with applications including free-form video retrieval and helping visually impaired people understand visual content.Large-scale datasets provide a common ground for video-to-language research [27].Source: §1 · p. 1
• The detector learns caption-word and video-region correlations from training data, using soft attention to trace consistent visual information across frames and predict concept words.It requires no external knowledge sources and can be trained end-to-end jointly with video-to-language models by adding its loss as an auxiliary term.Source: §1 · pp. 1–2
• Semantic attention selectively combines detected concepts with input and output words to improve encoder hidden-state representations and word prediction.The mechanism focuses on detected concept words during both language encoding and decoding.Source: §1 · p. 2

1.2. Contributions

• On LSMDC 2016, the approach achieves the best accuracies in fill-in-the-blank, multiple-choice test, and movie retrieval, with comparable performance in movie description.The evaluation covers all four tasks: fill-in-the-blank and multiple-choice test for QA and captioning, movie retrieval, and movie description.Source: §1 · p. 2, §1.2 · p. 3

1.1. Related Work

• Unlike prior video captioning work [21] and methods relying on external classifiers, this approach learns detection and caption generation jointly from caption data and extends semantic attention beyond [38] to video captioning, retrieval, and QA.The paper contrasts its training setup with approaches that use external datasets or separately learned classifiers.Source: §1.1 · p. 2

1.2. Contributions

• The work presents a plug-in approach for captioning, retrieval, and QA, claiming the first end-to-end model for both concept detection and language generation and the first semantic attention model for video-to-language tasks.It validates the approach across all four LSMDC 2016 tasks.Source: §1.2 · p. 2

2.1. Preprocessing

• The word dictionary contains 12,486 words appearing more than three times in the dataset, with embeddings trained using word2vec skip-gram [18] at dimension 300.Source: §2.1 · p. 3
• Videos are sampled at one frame per ten, with a maximum of 40 frames and wider intervals for longer videos.Source: §2.1 · p. 3
• Each frame’s ImageNet-pretrained ResNet features [12] [24] are reduced from a 7 × 7 × 2,048 map to a 4 × 4 × 500 spatial representation.The reduction uses 2 × 2 max-pooling followed by a 3 × 3 convolution, speeding detector training without significantly harming detection performance.Source: §2.1 · p. 3

2.2. An Attention Model for Concept Detection

• The detector selects the 2,000 most frequent caption nouns, verbs, and adjectives using NLTK on training captions [7], trains on caption-presence labels, and returns the top 10 words.Training uses sigmoid cross-entropy for multi-label predictions; inference selects the top K confidence scores.Source: §2.2 · pp. 3–4
• Tracing LSTMs use soft spatial attention to track regions consistently across frames, then fuse the resulting traces to discover concept words without ground-truth bounding boxes.The model uses one trace per spatial feature region, with 16 traces for a 4 × 4 grid; its attention is designed to follow temporally consistent meanings rather than treat frames independently.Source: §2.2 · p. 3

3. Video-to-Language Models

• CT-SAN combines task-specific base models with a shared concept word detector and semantic attention across four LSMDC tasks.The tasks are movie description, fill-in-the-blank, multiple-choice, and movie retrieval; the latter two models are detailed in the supplementary file.Source: §3 · p. 4

3.1. A Model for Description

• Captioning uses video-encoding and caption-decoding LSTMs with input attention to select concepts dynamically and output attention to weight concepts when generating each word.Training combines word negative log-likelihood, attention-weight regularization, and the concept-discovery loss.Source: §3.1 · pp. 4–5

3.2. A Model for Fill-in-the-Blank

• Fill-in-the-blank uses a bidirectional LSTM initialized from the video representation, with semantic attention strengthening input words and attending to concepts at the blank position.The model adds word prediction only at the blank token and applies output semantic attention once at that step.Source: §3.2 · p. 5

4.1. The LSMDC Dataset and Tasks

• LSMDC 2016 evaluates four tasks on 118,114 sentences and 118,081 clips sampled from 202 movies, following the challenge protocols.Movie description generates a sentence; fill-in-the-blank predicts a word; multiple-choice selects among five captions; retrieval ranks 1,000 candidate videos. The described test sets include 10,053 public-test examples for multiple-choice and 30,000 examples from 10,000 clips for fill-in-the-blank.Source: §4.1 · p. 6

4.2. Quantitative Results

• Our approach ranks first in four movie-description metrics and is described as comparable to state-of-the-art methods; CT-SAN significantly improves on Base-SAN.Base-SAN removes the concept detector and semantic attention from CT-SAN.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• The three-model CT-SAN ensemble outperforms all fill-in-the-blank participants and beats the single-layer LSTM/BLSTM and Base-SAN baselines.The ensemble averages output word probabilities from three independently trained identical models.Source: Table 1 · p. 7, §4.2 · p. 7
• 67.0%, up from 63.8% with a six-model ensemble, while CT-SAN ranks first in the multiple-choice test.The task uses video queries and five candidate captions; the results also attribute gains to the concept detector and semantic attention.Source: Table 2 · p. 7, §4.2 · p. 7
• CT-SAN achieves the best movie-retrieval performance with significant margins; its ensemble averages seven similarity matrices from six retrieval models and the multiple-choice CT-SAN model.Retrieval ranks a query sentence against candidate videos using Recall@k and median rank.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Qualitative Results

• Qualitative examples show detected concepts aligning with clips and guiding plausible descriptions or answers, while errors include near-miss words, scene misunderstandings, and missed rare motion cues.A generated clock-focused description differs from the ground truth but detects relevant concepts; retrieval examples capture water-related concepts yet miss subtle mouth movement.Source: §4.3 · pp. 7–8

5. Conclusion

• The proposed end-to-end detector generates concept words as semantic priors, which semantic attention selectively exploits in video-to-language models.Source: §5 · p. 8
• The approach improved video captioning, retrieval, and question answering, and won LSMDC 2016 fill-in-the-blank, multiple-choice test, and movie retrieval.Source: §5 · p. 8

A. Details of Video-to-Language Models

• Appendix A provides further details of the video-to-language models introduced in Section 3, including the multiple-choice model.Source: A. Details of Video-to-Language Models · p. 10, §A.1 · p. 10

A.1. A Model for Multiple-Choice Test

• For multiple-choice testing, the model scores a query video against five candidate sentences and selects the sentence with the highest score.Source: §A.1 · p. 10
• The model uses an LSTM video encoder to initialize a sentence LSTM, applies input semantic attention, and omits output attention because its output is a score.Source: §A.1 · p. 10
• Training uses a max-margin structured loss to score matched video-sentence pairs above incorrect pairs, with margin Δ = 1 in the experiments.Source: §A.1 · p. 10

A.2. A Model for Retrieval

• The model scores query–video pairs by learning a joint representation of text and video with Compact Bilinear Pooling (CBP) [8].The shared multimodal representation captures common features across the video and query encoders.Source: §A.2 · p. 10
• Video is represented by the final hidden state of its encoding LSTM, while query text is encoded through input semantic attention and an LSTM.The query representation is the final hidden state of the query encoding LSTM.Source: §A.2 · p. 10
• The CBP output is projected through Wp, processed by a maxout layer, and multiplied by Ws to produce the query–video score.The maxout layer yields non-sparse activations while mitigating overfitting.Source: §A.2 · p. 10
• A max-margin structured loss favors aligned video–sentence pairs over misaligned pairs, and test videos are ranked by their scores for each query.The margin is ∆=3 in the experiments; the final model ensembles multiple score matrices for better retrieval performance.Source: §A.2 · p. 10

B.1. Implementation Details

• All models use Adam [15] with initial learning rates from 10−4 to 10−5, image mirroring, batch shuffling each epoch, Xavier initialization, and two-layer LSTM/BLSTM networks with layer normalization [1] and dropout at rate 0.2.Layer normalization [1] and dropout with a rate of 0.2 are applied to reduce overfitting.Source: §B.1 · p. 11
• For fill-in-the-blank, multiple-choice, and retrieval, the concept detector starts from the movie-description model; new parameters are random, and the full model is trained end-to-end on the provided training set.Source: §B.1 · p. 11
• Movie description uses 101079 training, 9578 validation, 10053 test, and 7409 blind-test video-sentence pairs, plus the organizers’ Para-Phrase AD sentences for training.Source: §B.1 · p. 11
• Fill-in-the-blank uses the official training set from splits of 296961 training, 98483 validation, and 30350 test examples, with answer probabilities averaged across three differently initialized models.Source: §B.1 · p. 11
• Multiple-choice uses the movie-description split and four provided distractors; its ensemble averages score matrices from six independently trained model copies.The authors note that adding other negative sentences improved accuracy, but their experiments use only the four distractors.Source: §B.1 · p. 11
• Retrieval uses movie-description LSTM settings and dropout of 0.5 before maxout; six retrieval models and one multiple-choice model contribute averaged similarity matrices for 1000 sentences and 1000 clips.The final matrix combines alignment scores for all sentence-clip pairs.Source: §B.1 · pp. 11–12

• Additional experiments are presented to support the validity of the concept word detector and semantic attention models.Source: C. More Experimental Results · p. 12

C.1. On the Quality of Concept Words

• rand-SAN uniformly samples K = 10 words from V candidates, whereas NN-SAN uses the concept words of the closest training video by averaged ResNet features.Source: §C.1 · p. 12
• no-ATT-SAN replaces spatial attention with a two-layer LSTM fed average-pooled visual features, transforming the video representation into concept words without spatial attention.Source: §C.1 · p. 12
• CT-SAN outperforms no-ATT-SAN in all four tasks, while no-ATT-SAN beats Base-SAN and NN-SAN, indicating benefits from spatial and semantic attention.Source: §C.1 · p. 12
• rand-SAN performs much worse than Base-SAN, showing that poor concept-word quality can undermine performance even with semantic attention.Source: §C.1 · p. 12
• Decoupled CT-SAN performs worse than end-to-end CT-SAN, while NN-SAN words are not very helpful for training the semantic attention network.Source: §C.1 · p. 12

C.2. Ablation Study

• An additional semantic-attention ablation evaluates movie description and FITB, with results presented in Table 6.Source: §C.2 · p. 12

C.3. On the Number of Concept Words

• Changing K among 5, 10, and 20 detected concept words yields only marginal performance differences, while larger K increases training time and overfitting risk.The experiment evaluates CT-SAN.Source: §C.3 · p. 13

D. More Examples and Qualitative Results

• The visualized spatial attentions roughly capture high-level concepts, such as a blue car moving left to right.Figures 6–7 show spatial attention examples; Figure 6 includes sampled keyframes and attention maps from three tracing-LSTMs.Source: D. More Examples and Qualitative Results · p. 13, Figure 6 · p. 14, Figure 7 · p. 15
• Movie-description examples compare CT-SAN and no-ATT-SAN with baselines, while additional examples include positive descriptions and near-miss or wrong outputs.The examples show generated descriptions alongside detected concept words; Figure 10 identifies positive and near-miss or wrong cases.Source: Figure 8 · p. 16, Figure 9 · p. 17, Figure 10 · p. 18
• Task examples show positive and near-miss or wrong fill-in-the-blank and multiple-choice outputs, plus positive and negative retrieval rankings for a sample sentence.Retrieval figures show ranked clips against the ground-truth clip, including the query “SOMEONEs face contorts in anguish as she gazes at the comatose woman.”Source: D. More Examples and Qualitative Results · p. 13, Figure 11 · p. 19, Figure 12 · p. 20