Source-linked AI summary
Video In Sentences Out
Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, Zhiqi Zhang
TL;DR
The paper addresses how to generate rich sentential descriptions of video rather than only recognize action classes. It combines event recognition with object tracking, role assignment, posture information, and linguistic generation, obtaining 25.5% true single-best descriptions while using a corpus designed for sentence evaluation.
Problem
Existing event-recognition methods and video corpora do not adequately support rich descriptions requiring participant identities, event roles, relations, properties, and motion.
Method
The system uses HMM-based event classification with participant object tracks, role assignments, body-posture features, and rules that render verbs, nouns, adjectives, prepositional phrases, and adverbs.
Results
25.5% (191/749) of single-best video-sentence pairs were judged true.
Takeaways & Limitations
Rich video description requires event recognition that recovers participant tracks, their roles, and changing body posture, not just an action label.
Takeaways & Limitations
Many existing corpora are ill-suited to evaluating sentential descriptions, so the paper uses the Mind’s Eye corpus and excludes synthetic test video.
Abstract
from arXiv · showhide
We present a system that produces sentential descriptions of video: who did what to whom, and where and how they did it. Action class is rendered as a verb, participant objects as noun phrases, properties of those objects as adjectival modifiers in those noun phrases, spatial relations between those participants as prepositional phrases, and characteristics of the event as prepositional-phrase adjuncts and adverbial modifiers. Extracting the information needed to render these linguistic entities requires an approach to event recognition that recovers object tracks, the trackto-role assignments, and changing body posture.
1 INTRODUCTION
The paper presents a system that converts short videos into rich sentences describing actions, participants, object properties, spatial relations, and event characteristics. It argues that this requires recovering participant tracks, their event roles, and changing posture rather than relying only on conventional event-recognition cues.
- 1 INTRODUCTION: The system generates sentences that encode actions as verbs, participants as noun phrases, object properties as adjectives, spatial relations as prepositional phrases, and event characteristics as adjuncts or adverbs.Its vocabulary contains 118 words spanning these linguistic categories.
- 1 INTRODUCTION: Rich descriptions require identifying which detected objects participate in an event, assigning them event roles, and determining relations used to construct referring expressions.For example, describing a person relative to a bicycle requires both spatial relations and participant-role information.
- 1 INTRODUCTION: The paper motivates event recognition based on participant interactions and motion structure, while object classes and visual properties primarily support noun-phrase and adjective generation.
- 1 INTRODUCTION: Common event-recognition approaches can exploit corpus-specific image or video correlations that are artefacts rather than defining properties of verb meanings.Examples include associating diving with blue backgrounds or confusing basketball and volleyball because their courts look similar.
2 THE MIND’S EYE CORPUS
The Mind’s Eye year-one corpus was designed to evaluate sentential video descriptions, unlike many existing datasets that lack interacting participants or use labels poorly suited to sentence generation. The paper trains on C-D1 and tests on non-overlapping, real-video portions of C-E1 while excluding synthetic test videos.
- 2 THE MIND’S EYE CORPUS: Many existing video corpora are ill-suited to sentential description because they depict single-participant actions, irrelevant action distinctions, or large activities whose labels do not map naturally to sentences.Examples include datasets containing only verbs such as “The person jumped” and labels such as Basketball or TaiChi.
- 2 THE MIND’S EYE CORPUS: The Mind’s Eye year-one corpus was specifically designed to evaluate sentential description and includes separate development and evaluation corpora.The paper uses C-D1 for training and C-E1 for testing, restricted to non-overlapping evaluation videos.
- 2 THE MIND’S EYE CORPUS: The experiments use only the corpus’s recognition and description portions, excluding its gap-filling and anomaly-detection portions.
- 2 THE MIND’S EYE CORPUS: The test corpus excludes synthetically generated video; the resulting sets contain 3,480 training videos and 749 test videos.Videos are provided at 720p and 30 fps, with lengths ranging from 42 to 1,727 frames.
- 2 THE MIND’S EYE CORPUS: The corpus contains 48 nominal verbs, but polysemy, synonymy, component actions, and multi-constituent descriptions make the video-to-verb mapping non-one-to-one.
3 OVERALL SYSTEM ARCHITECTURE
The system combines detection-based tracking, body-posture features, and role assignment to recognize video events and generate their linguistic descriptions. It uses object and person tracks, temporal features, and HMMs to recover action classes and participant roles.
- 3.1 TRACKING: Detection-based tracking composes high-recall per-frame detections into temporally coherent tracks for 25 trained object models across the test videos.Detectors are biased toward few false negatives, tracks are augmented with KLT projections, and pruning removes unreliable tracks.
- 3.4 GENERATING SENTENCES: The architecture converts detected action classes and associated tracks into sentential descriptions using templates, noun mappings, modifiers, spatial phrases, and motion adjuncts.The vocabulary and templates support verbs, participant noun phrases, object properties, spatial relations, and event characteristics.
- 3.2 BODY-POSTURE CODEBOOK: Body-posture representation adds continuous normalized part displacements and discrete codebook indices to person-detection features.Part displacements and their temporal derivatives are computed from detector part structure, while vector quantization supplies the discrete codebook feature.
- 3.3 EVENT RECOGNITION: HMMs classify 48 action classes from single-track or two-track time series containing participant motion, posture, object identity, and pairwise spatial features.The models use separate one-track and two-track training sets and model discrete, linear, and angular features with corresponding output distributions.
- 3.3 EVENT RECOGNITION: Testing selects the highest-likelihood mapping of tracks to agent and patient roles, enabling the recognized event to determine subject and object participants.For each video and action class, likelihoods are thresholded to produce ROC evaluations against human judgments.
4 EXPERIMENTAL RESULTS
The sentence-generation evaluation uses Experiment III likelihoods to produce candidate descriptions for test videos. Human judges assess whether each generated sentence is true of the video and whether it describes a salient event.
- SENTENCE EVALUATION: Experiment III generates sentences for the three most-likely action classes per test video, and human judges rate truth and event salience.The evaluation uses key frames and generated sentences from four test videos to illustrate the assessment.
5 CONCLUSION
The paper presents rich video descriptions as requiring event participants, role assignments, motion, and properties, rather than only action labels. Its approach combines detection-based tracking with a body-posture codebook and demonstrates the approach on 749 videos.
- CONCLUSION: Rich sentential video descriptions require recovering event participants, their roles, motion, and properties, which common feature-based event-recognition approaches cannot determine.The conclusion contrasts these requirements with spatiotemporal bags of words, volumes, and tracked feature points.
- CONCLUSION: The proposed approach uses detection-based tracking and a body-posture codebook to recover information needed for rich descriptions, demonstrated on a corpus of 749 videos.The paper states that it is unaware of prior work generating descriptions of comparable richness.