Source-linked AI summary
Coherent Multi-Sentence Video Description with Variable Level of Detail
Anna Senina, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, Bernt Schiele
TL;DR
Existing video-description methods mainly produce single sentences at a fixed detail level, leaving coherent description of complex videos and controllable detail insufficiently addressed. The paper predicts semantic representations from video, generates language from them, enforces a shared topic across sentences, and adds hand-centric recognition and word-lattice generation. It reports improved coherence and recognition-related description quality, while analyzing three human-written detail levels to guide relevant-segment selection.
Problem
Existing approaches mainly generate single-sentence video descriptions at a fixed level of detail, rather than coherent descriptions of complex videos with variable detail.
Method
The paper predicts an intermediate semantic representation from video, generates language from it, enforces a consistent high-level topic across sentences, and uses hand-centric recognition with word-lattice generation.
Results
Human judges rate the multi-sentence descriptions as more readable, correct, and relevant than related work, while hand-centric features improve recognition of manipulated objects and probabilistic SMT input improves individual sentences.
Takeaways & Limitations
Human descriptions show that shorter outputs compress information according to the video topic, supporting relevant-segment extraction and level-specific language modeling.
Takeaways & Limitations
The human-description BLEU comparison is not fully comparable because the human description has one fewer reference.
Abstract
from arXiv · showhide
Humans can easily describe what they see in a coherent way and at varying level of detail. However, existing approaches for automatic video description are mainly focused on single sentence generation and produce descriptions at a fixed level of detail. In this paper, we address both of these limitations: for a variable level of detail we produce coherent multi-sentence descriptions of complex videos. We follow a two-step approach where we first learn to predict a semantic representation (SR) from video and then generate natural language descriptions from the SR. To produce consistent multi-sentence descriptions, we model across-sentence consistency at the level of the SR by enforcing a consistent topic. We also contribute both to the visual recognition of objects proposing a hand-centric approach as well as to the robust generation of sentences using a word lattice. Human judges rate our multi-sentence descriptions as more readable, correct, and relevant than related work. To understand the difference between more detailed and shorter descriptions, we collect and analyze a video description corpus of three levels of detail.
1. Introduction
The paper targets automatic video description that is typically limited to single sentences and fixed detail, proposing coherent multi-sentence descriptions with selectable detail levels. Its approach combines semantic representations, topic consistency, hand-centric recognition, and linguistic generation mechanisms.
- Existing video-description systems mainly generate single sentences for snippets at a fixed level of detail.
- The model predicts an intermediate semantic representation from video and generates natural-language sentences from that representation.
- A high-level topic, such as the dish being prepared, keeps sentences about different activities conceptually consistent.
- The paper collects and analyzes descriptions at three detail levels, then selects relevant video segments according to a predicted topic for shorter descriptions.
- A hand-centric object-recognition model improves recognition of manipulated objects, while linguistic cohesion supports fluent multi-sentence text.
2. Related Work
Prior work uses templates, retrieval, language models, graphical structures, and discourse constraints for image or video description. This paper differs by modeling a global semantic topic for many-sentence consistency and by targeting multiple detail levels.
- Image and video descriptions have been generated with manually defined templates, retrieved sentences, or novel sentences composed by language models.
- Prior image methods connect detected objects, model discourse and cohesion, or reduce redundancy, generally producing shorter multi-sentence outputs.
- The paper models a global semantic topic to maintain consistency across many sentences, unlike approaches producing only one to three sentences in most cases.
- Video methods have used causal action graphs, paraphrasing and merging, or periodic template sentences based on detected concepts.
- The authors report no prior computer-vision work addressing descriptions at different detail levels; their approach summarizes activities or objects rather than merely lowering prediction specificity.
3. Analysis of human-written video descriptions of different levels of detail
The authors collect aligned descriptions of TACoS cooking videos at three lengths and analyze which information humans verbalize. Shorter descriptions increasingly retain topic-relevant activities and objects while omitting fine-grained details.
- Data collection: The corpus contains about 2600 description triples for 127 cooking videos covering 26 dishes, with detailed, short, and single-sentence versions.
- Category analysis: Figure 2 measures the percentage of descriptions containing at least one word from each semantic category.
- Category analysis: Activities, food items, and people appear in almost all descriptions, whereas tools, utensils, appliances, attributes, quantities, and spatial information decline as descriptions shorten.
- Category analysis: Tools, utensils, and appliances nearly disappear from single-sentence descriptions, while detailed descriptions mention fine-grained activities and objects.
- Verb analysis: Longer descriptions favor specific activities such as cut or take, whereas single-sentence descriptions often use abstract verbs such as prepare, cook, and make.
- Food-item analysis: Short descriptions mention topic-relevant main ingredients such as potato or carrot and omit common ingredients such as water, salt, or spice.
4. Generating consistent multi-sentence video descriptions at multiple levels of detail
The method generates coherent descriptions at multiple detail levels by predicting a semantic representation for video snippets, enforcing a shared topic, and selecting topic-relevant segments for shorter outputs.
- Multi-sentence video descriptions: The video is segmented into snippets, each represented by visual descriptors and an intermediate semantic representation for sentence generation.The representation uses CRF nodes for snippet-level entities and a high-level topic node.
- Multi-sentence video descriptions: A fully connected CRF adds a topic node connected to all semantic nodes, with the topic inferred from the entire video rather than one snippet.The topic state is used to promote consistency across descriptions of different activities.
- Multi-sentence video descriptions: At test time, the model fixes the topic to its highest-scoring state before computing the conditional probability of snippet-level states.This makes all snippets condition on the same predicted topic.
- Multi-level video descriptions: Short descriptions are produced by ranking snippets with tfidf scores that measure how discriminative their predicted semantic states are for the predicted topic.The method sums node-state tfidf scores and selects the highest-scoring segments.
- Multi-level video descriptions: The selected segments support short and single-sentence descriptions, while separate translation models can target the desired detail level.The paper also compares this strategy with extracting sentences from detailed descriptions and with a retrieval baseline.
5. Improving Visual features
The paper improves visual recognition by replacing semantically ambiguous attribute features with semantic unaries and by focusing recognition on left and right hands, which improves hand detection.
- Motivation: Noisy visual recognition is identified as a main limitation of the translation approach and is especially problematic for inferring the dish topic from food objects.The paper therefore targets semantic-representation quality and challenging ingredient recognition.
- Visual feature improvements: The revised CRF uses semantic unaries and hand-focused features instead of relying directly on generic dense-trajectory attribute representations.The hand-centric strategy is intended to improve object recognition in activity videos.
- Semantic unaries: Semantic attribute classifiers distinguish the same visual item by role, such as knife-TOOL versus knife-OBJECT, reducing ambiguity in CRF predictions.Each node/state classifier score can be used directly as a feature for its corresponding unary.
- Semantic unaries: Using one scalar weight per node-state classifier yields more discriminative unaries while reducing the model's number of parameters.The topic-node unary is defined similarly using composite activity-recognition features for the whole video.
- Hand-centric features: The hand-centric detector separately models left and right hands, extracts hand-region descriptors, and integrates detections into upper-body pose estimation.Separate components reflect that the hands can perform different roles in activities.
- Hand-centric features: Hand detection on 1277 Pose Challenge test images significantly improves over FPS, improves further when integrated into the pose model, and surpasses another hand detector.The comparison is reported for detection accuracy at varying distances from ground-truth positions.
6. Generating natural descriptions
The system translates semantic representations into sentences and uses probabilistic decoding plus post-processing to make descriptions more robust and cohesive.
- Parallel SR–sentence pairs train SMT to generate a sentence from a predicted semantic representation.
- Probabilistic input for SMT: Word lattices encode multiple visual-recognition hypotheses so SMT can use confidence scores rather than only the most probable SR.The lattice is a directed acyclic graph constructed from predicted ACTIVITY, TOOL, INGREDIENT, SOURCE, and TARGET nodes.
- Creating cohesive descriptions: The system post-processes independently generated sentences by fixing punctuation, merging related adjacent sentences, and adding referring expressions.Sentences are merged when they share a verb but differ in objects, or share an object but differ in verbs.
- Creating cohesive descriptions: Post-processing converts list-like outputs into more cohesive descriptions by combining sentences about related objects or actions.For example, separate knife and peeler sentences become one sentence, while separate peeling and slicing sentences are combined.
7. Evaluation
The evaluation measures semantic-recognition accuracy, generated-description quality, and performance across detailed, short, and single-sentence settings. The proposed recognition, probabilistic SMT, and post-processing components improve reported quality, although a gap to human descriptions remains.
- Visual Recognition: Handled-object recognition is the most difficult semantic-representation node, reaching 33.2% accuracy versus at least 60.8% for the other nodes.The passage attributes this to the larger number of states and high intra-class variability of ingredients.
- Visual Recognition: Adding semantic unaries improves activity accuracy by 5.6% and object accuracy by 3.9% over the prior system.
- Visual Recognition: Hand-centric color Sift features improve object accuracy by 6.7% and dish accuracy by 9.9% in the reported comparison.Compared with, the overall improvements are 5.3% for ACTIVITY, 2.5% for TOOL, 15.9% for OBJECT, 0.7% for SOURCE, and 2.8% for TARGET.
- Multi-sentence generation: With ground-truth intervals, the improved semantic representation raises BLEU@4 by 1.9% per sentence and 8.1% per description, while human ratings also increase.Readability rises by 0.8, correctness by 0.3, and relevance by 0.2 relative to.
- Multi-sentence generation: Word-lattice SMT increases all reported scores, including a 2.3% BLEU@4 gain and a 0.3 readability gain.The method can select a less likely visual configuration when the language model favors it.
- Multi-sentence generation: Linguistic post-processing further improves readability to 3.8, although the system remains below human descriptions.The comparison concerns detailed-description generation and the reported human-evaluation scale.
- Multi-level generation: For short descriptions, selecting relevant segments captures important video events, while single-sentence generation performs best when trained on short descriptions.The short-description analysis reports higher or similar human-judgment scores than using human-selected short-description intervals.
8. Conclusion
The paper advances coherent multi-sentence video description by combining topic consistency, hand-centric visual features, and probabilistic language generation. It also supports multiple detail levels through corpus analysis and topic-guided segment selection.
- Inferring the high-level topic helps ensure consistency across sentences in multi-sentence video descriptions.
- Hand-centric features improve recognition of manipulated objects, leading to consecutive improvements in generated descriptions.
- Probabilistic input in statistical machine translation improves sentence quality for individual video snippets.
- A corpus of human descriptions at three detail levels reveals how verbalized information changes as descriptions become shorter.
- As descriptions shorten, information is compressed according to the video topic; selecting the most relevant segments and matching language-model detail improves generation.