Source-linked AI summary
The Long-Short Story of Movie Description
Anna Rohrbach, Marcus Rohrbach, Bernt Schiele
TL;DR
Movie description on datasets such as MPII-MD is challenging, and the difficulties of existing methods have not been analyzed in detail. The paper trains visual classifiers for verbs, objects, and places from weak sentence annotations, then uses their scores as LSTM input; it reports the highest performance on MPII-MD across automatic measures and extensive human evaluation.
Problem
Movie-description datasets are challenging for visual recognition and automatic description, and the task's difficulties lacked detailed analysis.
Source: §1 · p. 1Method
The paper trains robust classifiers for verbs, objects, and places from weak sentence annotations and uses their scores as input to an LSTM.
Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 3Results
The approach achieves the highest performance on MPII-MD across automatic evaluation measures and extensive human evaluation.
Source: §6 · p. 13Takeaways & Limitations
Performance across methods correlates with sentence and clip difficulty, while the data show biases toward human subjects and verbs like “look.”
Source: §6 · p. 13
Abstract
from arXiv · showhide
Generating descriptions for videos has many applications including assisting blind people and human-robot interaction. The recent advances in image captioning as well as the release of large-scale movie description datasets such as MPII Movie Description allow to study this task in more depth. Many of the proposed methods for image captioning rely on pre-trained object classifier CNNs and Long-Short Term Memory recurrent networks (LSTMs) for generating descriptions. While image description focuses on objects, we argue that it is important to distinguish verbs, objects, and places in the challenging setting of movie description. In this work we show how to learn robust visual classifiers from the weak annotations of the sentence descriptions. Based on these visual classifiers we learn how to generate a description using an LSTM. We explore different design choices to build and train the LSTM and achieve the best performance to date on the challenging MPII-MD dataset. We compare and analyze our approach and prior work along various dimensions to better understand the key challenges of the movie description task.
1 Introduction
Movie description is studied on large-scale datasets such as MPII-MD, where visual recognition and automatic description are challenging. The paper examines existing methods and presents robust visual classifiers, LSTM design choices, and an analysis of task challenges.
1 Introduction
- MPII-MD and M-VAD enable study of movie description, a challenging task with lower performance than simpler video datasets such as MSVD.The introduction notes that a detailed analysis of the task's difficulties was missing.Source: §1 · p. 1
- The paper builds classifiers for verbs, objects, and places from weak sentence annotations, then evaluates LSTM design choices for generating descriptions.The authors report outperforming related work on MPII-MD in automatic and human evaluation and analyze task challenges.Source: §1 · p. 1
2 Related Work
Prior work addresses image and video description with recurrent networks, while large movie-description datasets pose a broader and more varied challenge. This work builds on movie-description methods while distinguishing its handling of weakly supervised labels and LSTM design.
Image captioning
- Image-description methods commonly use RNNs, especially LSTMs, and prior work has analyzed generated-description novelty and nearest-neighbor baselines.The passage names Flickr30k and MS COCO Captions as image-captioning datasets.Source: §2 · p. 2
Video description
- Video-description research spans semirealistic, small-scale, and constrained settings, with methods combining CNN features, LSTMs, and joint visual-semantic models.Several approaches describe a short clip with a single sentence.Source: §2 · p. 2
Movie description
- Movie datasets MPII-MD and M-VAD cover broader domains than earlier video-description datasets, but their weak sentence annotations provide no additional labels.Prior movie-description approaches include attention-based and encoder-decoder models; the paper says its generation approach is closest to.Source: §2 · p. 2
- The approach trains classifiers for verbs, objects, and places, then feeds selected robust-classifier scores to an LSTM.The visual features include dense trajectories, object-detector responses, and Places-CNN scores.Source: Figure 1 · p. 3
3 Approach
The approach first performs visual recognition with classifiers tailored to label semantics and visuality, then generates descriptions with an LSTM. It handles weak sentence annotations by selecting robust visual labels and explores LSTM architectures and training choices.
3 Approach
- The approach has two stages: visual recognition with classifiers tailored to label semantics and visuality, followed by LSTM-based sentence generation.The authors examine design choices for building and training the LSTM.Source: §3 · p. 3
3.1 Visual Labels for Robust Visual Classifiers
- Labels are parsed from weak sentence annotations, and failed parses are matched to initial labels to retain potential training labels.The approach selects labels considered visual and robustly recognizable rather than keeping every noisy initial label.Source: §3.1 · p. 3
- Verbs, objects, and places are treated as separate semantic groups, with group-specific visual representations and one-vs-all classifiers.The features are Improved Dense Trajectories for verbs, LSDA responses for objects, and Places-CNN scores for places; separate training aims to discard wrong negatives.Source: §3.1 · pp. 3–4
- Candidate labels are matched to verb, place, or object classes, while labels outside those groups or not learnable by classifiers are discarded.Learnability is assessed using a minimum area-under-the-ROC-curve requirement.Source: §3.1 · p. 4
3.2 LSTM for Sentence Generation
- At each time step, the LSTM receives visual-classifier scores and the previous word, and generates a word using a learned word embedding.The paper uses the decoder variant with visual probabilistic predictions, which found to provide the richest visual input.Source: §3.2 · p. 4
- The LSTM experiments compare one- and two-layer designs, including a factored two-layer variant, and vary dropout placement and ratio.Dropout may be applied to language input, visual input, both inputs, or the LSTM output.Source: §3.2 · pp. 4–5
- Training comparisons include step-based learning-rate halving with validation-selected settings and a continuously decreasing polynomial strategy.The polynomial strategy is compared with the default step-based schedule.Source: §3.2 · p. 5
4 Evaluation
The evaluation tests visual-classifier and LSTM design choices on MPII-MD, then compares the best system with prior work. Semantic label selection and classifier training, along with LSTM training choices, improve performance; the final system leads prior methods on automatic measures, with mixed human-evaluation results.
4.1 Analysis of our approach
- The evaluation trains SVMs on 56,861 clips and selects labels appearing at least 30 times, yielding 1,263 labels for validation on 4,930 clips using METEOR.The prior-work comparison evaluates 6,578 test clips with automatic measures and human rankings from 1,300 snippets.Source: §4.1 · p. 5, §4.2 · p. 8
- Selecting visual labels and matching classifier training and features to semantic groups improves performance while reducing the LSTM input to 263 labels.Group-specific one-vs-all SVMs avoid “wrong” negatives; using the same feature for all 263 selected labels performs worse.Source: Robust visual classifiers · p. 6, Table 1 · p. 6
- A one-layer LSTM with dropout after the LSTM performs best among the tested dropout placements, and a dropout ratio of 0.5 works best.The one-layer and two-layer unfactored architectures perform equally well, while the two-layer factored architecture is inferior.Source: LSTM architectures · p. 7, Table 2 · p. 7
- Step-based learning outperforms polynomial learning, and a three-network ensemble often improves METEOR over single networks.With step size 4000, iteration 15,000 gives the best overall performance; the authors identify learning rate and step size, dropout, METEOR-based iteration selection, and ensembles as important for reducing over-fitting and improving generation.Source: Table 3 · p. 7, LSTM architectures · p. 8, Table 4 · p. 8
4.2 Comparison to related work
- The proposed system improves all automatic measures over on MPII-MD test data, including by 1.44 METEOR points, and also improves over [33].Human evaluation favors it over both prior works for Correctness and Relevance, but favors S2VT [33] for Grammar; the proposed system generates longer sentences (8.7 versus 7.4 words).Source: §4.2 · pp. 8–9, Table 5 · p. 9
- The proposed system produces a vocabulary of 605 words, compared with 94 for and 86 for [33], against 6,422 unique words in the test set.The authors attribute its more varied outputs to diverse, robust visual classifiers and LSTM choices that better match words to classifier scores.Source: §4.2 · p. 9
5 Analysis
The section compares three methods on MPII-MD to examine where they succeed and fail. Performance varies with sentence and clip difficulty, verb topics, and sentence characteristics such as subject type.
5.1 Difficulty versus performance
- All methods’ METEOR scores decline as sentence length increases or average word frequency decreases; ours most notably outperforms the others as difficulty increases.The correlation is stronger for word frequency than for sentence length.Source: §5.1 · pp. 10–11, Figure 4 · p. 10
- Our method is best throughout the textual-difficulty range, especially for harder sentences; visual kNN difficulty also correlates with all methods’ performance.Visual kNN ranks reference sentences using the best METEOR score among k = 10 visually nearest training clips.Source: §5.1 · p. 11, Figure 5 · p. 11
- Textual difficulty has the strongest correlation with method performance, while sentence length has the weakest.Visual difficulty also strongly correlates with performance; all methods perform better on shorter, common sentences, and ours notably wins on longer sentences.Source: Figure 4 · p. 10, §5.1 · p. 11, Figure 5 · p. 11
5.2 Semantic analysis
- Our method is best across WordNet verb topics except “communication,” where wins.The communication topic’s frequent verbs include “look up” and “nod”; the best-performing “cognition” topic is biased toward “look at.”Source: Figure 6 · p. 12, §5.2 · p. 12
- “Motion” and “contact” topics remain challenging amid diverse verbs; less diverse, frequent “perception” verbs score better, while abstract verbs tend to score lower.The topic analysis uses WordNet senses, which may be noisy.Source: §5.2 · pp. 12–13, Table 6 · p. 12
- Frequent visual verbs such as “look” receive higher scores, while sentences without verbs or subjects, or with non-human subjects, score lower.The authors attribute the latter pattern in part to the dataset’s bias toward “Someone” as a subject.Source: §5.2 · p. 13
6 Conclusion
The paper combines semantic-group visual classifiers with an LSTM for movie description and reports the highest MPII-MD performance across automatic measures and extensive human evaluation. Its analysis identifies data biases and difficulty patterns that motivate future work on less frequent words and less visual descriptions.
6 Conclusion
- Visual classifiers distinguish verbs, objects, and places from weak sentence annotations, and their scores feed an LSTM for description generation.The classifiers are trained discriminatively with potentially noisy negatives removed, and the method selects a small set of the most reliable classifiers.Source: §6 · p. 13
- The approach achieves the highest MPII-MD performance across all automatic evaluation measures and extensive human evaluation.Source: §6 · p. 13
- The analysis finds performance associated with frequent words, sentence simplicity, visual verbs, and textual and visual difficulty, alongside a bias toward human subjects and “look”-like verbs.Source: §6 · p. 13
- Future work should address less frequent words and less visual descriptions, potentially using external text, audio and dialogue, and context across multiple sentences.Source: §6 · p. 13