PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Video Description

Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, Marcus Rohrbach

arXiv:1812.06587v2cs.CV

TL;DR

Video description models can generate plausible but ungrounded sentences, motivating methods that connect language to visual evidence. The paper introduces noun-phrase box annotations and a grounded generation model, achieving state-of-the-art results across video and image description while improving grounding.

  • Problem

    Video description models often produce plausible language that is not grounded in the video, raising concerns about hallucination, bias, accountability, and trustworthiness.

  • Method

    ActivityNet-Entities links each concrete noun phrase to the corresponding referred instance’s bounding box in one video frame, enabling grounded training and evaluation.

  • Results

    The paper achieves state-of-the-art performance on video description, video paragraph description, and image description, with improved grounding and visual explainability.

  • Takeaways & Limitations

    Explicit box supervision improves generated-description accuracy and explainability across the paper’s video and image description experiments.

  • Takeaways & Limitations

    Video Event Description assumes temporal event boundaries are given and therefore does not address temporal localization.

Abstract

from arXiv · show

Video description is one of the most challenging problems in vision and language understanding due to the large variability both on the video and language side. Models, hence, typically shortcut the difficulty in recognition and generate plausible sentences that are based on priors but are not necessarily grounded in the video. In this work, we explicitly link the sentence to the evidence in the video by annotating each noun phrase in a sentence with the corresponding bounding box in one of the frames of a video. Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase. This allows training video description models with this data, and importantly, evaluate how grounded or "true" such model are to the video they describe. To generate grounded captions, we propose a novel video description model which is able to exploit these bounding box annotations. We demonstrate the effectiveness of our model on our dataset, but also show how it can be applied to image description on the Flickr30k Entities dataset. We achieve state-of-the-art performance on video description, video paragraph description, and image description and demonstrate our generated sentences are better grounded in the video.

1. Introduction

Video description models can produce plausible but ungrounded language, making accountability and reliable human interaction difficult. ActivityNet-Entities addresses this by linking noun phrases to video-frame boxes and enabling grounded generation and evaluation.

  • Motivation: Video descriptions may hallucinate objects or fail to identify which visual instance a noun phrase refers to, limiting accountability and follow-up interactions.Grounding is especially important for applications such as answering questions about people in a video.
  • Dataset: ActivityNet-Entities links noun phrases to the specific bounding box of the referred instance in a video frame.The dataset contains 51.8k annotated video segments/sentences and 157.8k labeled bounding boxes, with sparse single-frame annotations.
  • Dataset: The dataset supports both training models to use visual evidence and evaluating grounding for generated words or phrases.Annotations target the referred instance rather than every object sharing its category.
  • Method: The proposed model jointly generates words and refines grounding, using explicit box supervision to improve description generation over unsupervised methods.The framework is designed to compare grounding-supervised generation with approaches that use region features without penalizing grounding.
  • Scope: The framework is demonstrated for grounded video descriptions and applied to image captioning on Flickr30k Entities.The paper reports improvements in generated captions and grounding quality on both video and image description tasks.

2. Related Work

Prior description systems evolved from template-based pipelines to neural attention and detector-based methods. The paper addresses video-specific annotation challenges with noun-phrase grounding and sparse region supervision.

  • Video & Image Description: Template-based captioning fills predefined slots with detected visual evidence but remains restricted by its template structure.Neural and attention-based methods later became dominant on major captioning benchmarks.
  • Video & Image Description: Object-detector-based methods separate visual recognition from sentence generation, using region attention or region labels and templates.Off-the-shelf proposals can be biased toward source-dataset classes rather than target-description content.

3. ActivityNet-Entities Dataset

ActivityNet-Entities augments ActivityNet Captions with noun-phrase-level bounding boxes, enabling explicit grounding supervision and evaluation for video description. Annotations prioritize precise referents while sparsely sampling frames to maintain scale.

  • Motivation: ActivityNet Captions provides dense language annotations but lacks the grounding annotations needed to train and test explicit video grounding.ActivityNet-Entities adds entity-level bounding boxes to address this gap.
  • Dataset scope: 158k noun-phrase bounding boxes across 14,281 annotated videos make ActivityNet-Entities a large-scale resource for explicitly grounded video description.The dataset contains about 52k video segments with at least one noun-phrase annotation.
  • Annotation design: To favor scale over density, each noun phrase is annotated in only one frame per segment, reflecting the cost of dense frame-level region annotation.Noun phrases may denote a single instance or a group and can include modifiers, pronouns, or prepositions.
  • Annotation process: Annotators identify concrete noun phrases and draw boxes in one clearly showing frame from 10 uniformly sampled frames per video segment.Annotations include coreference resolution and multi-instance boxes; 10% of final annotations refer to multi-instance boxes.
  • Dataset representation: Noun-phrase annotations are converted into 432 word-level object classes using nouns and pronouns whose training and validation frequency reaches 50.The paper models sentence generation at the word level despite collecting noun-phrase annotations.

4. Description with Grounding Supervision

The framework jointly generates descriptions and grounds their words through grounding, region attention, and language-generation modules. It combines language, attention, classification, and localization supervision over region proposals.

  • Framework overview: The framework contains grounding, region attention, and language-generation modules that detect visual clues, attend to regions, and decode words.The language model dynamically attends to video frames or regions for visual clues.
  • Joint supervision: The joint objective combines language-generation, region-attention, region-classification, and object-grounding cross-entropy losses.The three grounding-related losses are weighted by coefficients selected on the validation split.
  • Region attention: At each generation step, region attention assigns weights to fine-grained object regions and feeds the attended encoding, with temporal attention, into the language LSTM.Unlike temporal attention, region attention operates on object regions rather than whole frames.
  • Supervised attention: Supervised attention regresses region attention weights toward indicators for regions whose IoU with the ground-truth box exceeds 0.5.This supervision encourages attention to the correct region when generating a visually groundable word.
  • Grounding module: The grounding module estimates object-class probabilities for regions and can use region classification or sentence-conditioned grounding objectives.Classifier initialization transfers detection knowledge from a Visual Genome detector through embedding-space nearest neighbors.
  • Grounding module: Region proposals are encoded with visual features, region-class similarity, and normalized spatial-temporal location before self-attention produces grounding-aware region encodings.These encodings are then passed to the region attention module.

5. Experiments

Experiments evaluate grounded video description on ActivityNet-Entities and transfer the framework to Flickr30k Entities image description. Grounding supervision improves localization and captioning, while GVD achieves strong benchmark results across video, paragraph, and image description.

  • Datasets: Experiments use ActivityNet-Entities for video event and paragraph description, and Flickr30k Entities to test image-description transfer.COCO captioning is excluded because its caption words do not exactly match its 80 object annotations.
  • Compared methods: Experiments compare against state-of-the-art video, paragraph, and image description methods using matched feature settings and three-run average scores.Video baselines include Masked Transformer and Bi-LSTM+TempoAttn; paragraph and image baselines include MFT, NBT, and BUTD.
  • Evaluation setting: 82.5%/83.4% localization upper bounds on ActivityNet-Entities validation/test sets constrain grounding evaluation to the selected region proposals.The paper assumes ground-truth temporal event boundaries and evaluates the language-generation component.
  • Video event description: 1–13% relative validation gains across all metrics and an 8.8% Bleu@4 test improvement favor supervised grounding methods over unsupervised variants.Test-set differences for Bleu@1, METEOR, CIDEr, and SPICE remain within ±2%.
  • Grounding ablations: 43.5% grounding accuracy versus 22.3% with box supervision shows that supervision improves object localization, while classification loss is needed to maintain classification accuracy.Classification loss alone can also implicitly learn grounding and retain fair grounding accuracy.
  • Video event description: GVD sets new state of the art on ActivityNet Captions for Bleu@1, METEOR, and SPICE, with relative gains of 2.8%, 3.9%, and 6.8%.It is slightly worse on Bleu@4 and CIDEr, while self-attention helps supervised methods but hurts unsupervised ones.
  • Video paragraph description: The model outperforms the video-paragraph-description state of the art by a large margin, despite generating each event description independently.The authors hypothesize that temporal attention models event context through the Bi-GRU context encoder.
  • Image description: GVD sets new state of the art on all five Flickr30k Entities metrics with relative gains up to 10%, alongside improved localization and region classification.The supervised method exceeds the unsupervised baseline by 1–3.7% across metrics.

6. Conclusion

The paper introduces ActivityNet-Entities and a grounded video description model, showing improved description quality, explainability, and grounding across video and image description tasks.

  • ActivityNet-Entities enables joint study of video description and grounding through noun-phrase bounding-box annotations.The dataset also supports evaluating how well generated sentences are grounded.
  • Bounding-box supervision improves description accuracy and explainability by linking generated words to corresponding regions.The model points to evidence in video frames or images while generating descriptions.
  • The model achieves state-of-the-art description quality on ActivityNet-Entities for both sentence- and paragraph-level evaluation, with significantly improved grounding.
  • On Flickr30k Entities, the adapted model outperforms existing image-description methods.

A. Appendix

The appendix supplies supplementary dataset information, metric clarification, additional experiments, Flickr30k Entities results, and implementation details.

  • The appendix provides additional details, evaluations, and qualitative results.
  • Dataset supplements cover the annotation interface and examples, while another section clarifies four localization metrics.
  • Additional experiments include ActivityNet-Entities ablations and qualitative results, Flickr30k Entities results, and implementation details such as training procedures.

A.1. Dataset

The dataset defines noun phrases as short, non-recursive expressions that refer to specific visual regions and can be grounded with bounding boxes.

  • Noun phrases include determiners, prepositional phrases, and other phrases referring to the same bounding concept and box.Examples include “a piece of exercise equipment,” “the woman on the right,” and “a shirt of red color.”
  • Complex noun phrases are split into short, non-recursive phrases averaging 2.23 words.For example, “the man in a white shirt with a heart” is divided into three phrases.
  • Annotatable noun phrases must refer to a specific image region that can be represented by a bounding box.
  • Supported references include single instances, multiple instances, groups, scenes or regions, pronouns, and adjectives.

Annotator instructions

Annotation instructions specify phrase coverage, handling of multiple instances and co-reference, exclusions, quality control, and the dataset’s object-class inventory.

  • Each caption word may belong to at most one noun phrase, preventing overlapping annotations for nested phrases.
  • Multiple boxes are annotated when a noun phrase refers to multiple instances, with special handling for groups or difficult separation.More than five instances are marked as a single group box; five or fewer may use one box when strong occlusion prevents separation.
  • Abstract or scene-absent noun phrases are not annotated, while one box may support multiple co-referential noun phrases.
  • Annotations underwent daily inspection, annotator feedback, unclear-example flagging, and postannotation verification.
  • The approach uses 432 object classes thresholded at 50 occurrences, while annotations retain full noun phrases without thresholds.

A.2. Localization Metrics

The paper evaluates localization in both ground-truth and generated sentences, distinguishing joint object-word correctness from localization quality among correctly predicted objects. Predictions count as correct only when the object word is correctly predicted and its region overlaps the ground-truth box by more than 0.5 IoU.

  • Metric setup: Attn. and Grd. measure attention and grounding localization accuracy on ground-truth sentences during inference.F1all and F1loc instead evaluate generated sentences.
  • Metric setup: A region prediction is correct only when the object word is correct and its predicted region has IoU > 0.5 with the ground-truth box.
  • F1all: F1all combines object-word prediction and localization, using precision and recall over correctly predicted and localized words.The metric is defined using generated and ground-truth object-word counts together with correctly predicted-and-localized words.
  • F1loc: F1loc measures localization among correctly predicted object words, ignoring language-generation errors because not every scene object has a box annotation.
  • Aggregation: Scores are computed per object class, with zero assigned when a class is never predicted, then averaged over object classes in each split.Only the first instance is considered when multiple instances of an object occur in the target sentence.

A.3. Results on ActivityNet-Entities

The experiments compare supervised and unsupervised grounding approaches across video and image description settings, including attention-module ablations and qualitative grounding examples. The supervised methods outperform unsupervised baselines on the reported Flickr30k Entities validation metrics, while the video paragraph model exceeds the compared state-of-the-art method despite an unfavorable split comparison.

  • Ablation study: The two attention modules perform similarly alone, but their combination performs best, indicating complementary contributions.
  • Qualitative results: Qualitative video examples show the supervised model grounding object words such as “man,” “Christmas tree,” “saxophone,” and “horses,” while baselines sometimes hallucinate objects.
  • Image description: 0.9-4.8% relative improvement across all metrics is reported for the best supervised method over the best unsupervised baseline on Flickr30k Entities validation data.The exception is Sup. Cls., which has a slightly inferior CIDEr result.
  • Qualitative results: Qualitative image examples show the supervised model attending to “man,” “dog,” and “snow,” while unsupervised or BUTD systems miss some of these words.
Loading 1812.06587v2…