Source-linked AI summary
Connecting Vision and Language with Localized Narratives
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, Vittorio Ferrari
TL;DR
Existing captioning datasets often provide sparse or coarse visual grounding, leaving many words and relations unlocalized. Localized Narratives synchronizes speech, text, image regions, and mouse traces to ground every word, and the authors scale it to 849k images while demonstrating controlled captioning. The authors report that the annotations are rich, diverse, accurate, and efficiently produced, with an author-supported limitation arising from imperfect temporal coordination by annotators.
Problem
Existing image-captioning datasets provide limited visual grounding, while deeper grounding requires linking specific words, including relations, to image regions.
Method
Annotators describe images aloud while hovering over described regions; speech, transcription, image, and mouse traces are synchronized to produce word-level grounding.
Results
The dataset covers 849k images, and the authors report rich, diverse, accurate, and efficient annotations demonstrated through controlled image captioning.
Takeaways & Limitations
Synchronized word-level grounding supports controlled captioning and provides a publicly available multimodal resource for applications using image, text, speech, and mouse traces.
Takeaways & Limitations
Some trace points fall outside object boxes because annotators circle objects or begin moving before or after describing them.
Abstract
from arXiv · showhide
We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.
1 Introduction
Localized Narratives address sparse visual grounding by synchronizing spoken descriptions with mouse traces, localizing every word while supporting efficient, large-scale annotation and controlled captioning.
- Motivation and proposal: Localized Narratives synchronize spoken descriptions with mouse traces, yielding a trace segment that grounds every word to an image location.The protocol grounds nouns, relations, and attributes through synchronized voice and pointer movement.
- Scale and analysis: 144.7 seconds is the total average annotation time, including 40.4 seconds for speaking and pointing and 104.3 seconds for manual transcription.The authors report this as lower-cost than related grounded captioning datasets requiring more complicated annotation and manual bounding boxes.
- Scale and analysis: 848,749 images across COCO, ADE20K, Flickr30k, and Open Images were annotated and released publicly.The collection covers the complete COCO, ADE20K, and Flickr30k datasets, plus 671k Open Images images.
- Scale and analysis: The dataset is analyzed as rich, diverse, accurate, and efficient, with grounding for nouns, verbs, prepositions, and other word types.Its captions are substantially longer than those in most previous datasets, while spatial indications also vary across annotations.
- Applications: Controlled image captioning uses mouse traces as fine-grained control signals for selecting which image regions to describe and in what order.The paper presents this as an initial application of the four synchronized modalities and as a potentially assistive interface.
- Contributions: The paper’s contributions are a new word-level grounded annotation format, annotations for 848,749 images, analysis of the data, and controlled image captioning.The protocol connects image, speech, text, and mouse-trace modalities.
2 Related Work
Prior captioning datasets provide whole-image, noun-level, or phrase-level grounding, whereas Localized Narratives grounds every word with synchronized mouse traces using common input devices.
- Captioning datasets: Classical image captioning and visual paragraph generation associate a whole caption with an entire image rather than grounding specific words or regions.The lack of proper grounding was identified as problematic for deeper vision-language connections.
- Captioning datasets: Flickr30k Entities grounds nouns, including attached adjectives, to bounding boxes, while Visual Genome provides short region-level phrases.Visual Genome can seed scene graphs, but its region descriptions do not individually ground every word.
- Captioning datasets: Localized Narratives grounds every word, including spatial and relational terms, to a specific image region represented by its trace segment.This differs from noun-level and phrase-level grounding in earlier datasets.
- Annotation protocols: Earlier grounded datasets require manually drawing many bounding boxes after captioning, making annotation more unnatural and time-consuming than the Localized Narratives protocol.The comparison is stated for Flickr30k Entities and Visual Genome.
- Annotation protocols: SNAG used voice and specialized gaze-tracking hardware but produced annotations for only 100 images because its setup was expensive and complicated.Localized Narratives instead uses a mouse, keyboard, and microphone, which are commonly available.
- Voice-based annotation: Related voice-based work includes object-instance class-and-box annotation, whereas Localized Narratives produces long free-form captions with synchronized mouse traces.Other cited efforts include voice-annotated and time-aligned video actions.
3 Annotation Process
Localized Narratives combines synchronized speech and mouse movements with accurate manual transcription to produce word-level visual grounding. The annotation process uses alignment and quality control to connect each manually transcribed word to a mouse-trace segment.
- Data collection: Annotators simultaneously describe visible image content with their voice and point to the described regions using the mouse.Instructions cover objects, attributes, and relations, including moving between two objects while describing their relationship.
- Transcription: The protocol combines timestamped but error-prone ASR output with accurate manual transcription to obtain captions that are both accurate and synchronized.Annotators manually transcribe their recordings immediately after narration; sequence-to-sequence alignment transfers timestamps from the ASR transcription.
- Transcription alignment: Dynamic alignment matches automatic and manual words while preserving word order, using edit distance to select the optimal matching.The matching assigns timestamps from automatic words to manual words, including cases where a manual word has no direct match.
- Timestamp transfer: Matched manual words inherit the time span of their aligned automatic words, while unmatched words receive intervals between neighboring matches.These intervals are transferred to the mouse trace to define one trace segment for each manually transcribed word.
- Quality control: The process uses the optimal alignment distance as an automatic quality-control metric and separately evaluates semantic, visual-grounding, and transcription accuracy.A high alignment distance indicates a large discrepancy between the two transcriptions, potentially reflecting transcription or ASR errors.
4 Dataset Collection, Quality, and Statistics
Localized Narratives were collected at scale with an efficient protocol and evaluated as rich, diverse, accurate, and densely grounded image descriptions. Manual checks found high semantic and transcription accuracy, while trace styles and occasional timing offsets define important quality considerations.
- Annotation cost: 144.7 seconds per image was the average annotation cost, including 40.4 seconds for narration and 104.3 seconds for manual transcription.This was reported as 4.4× faster than Flickr30k Entities’ 643-second process, which did not include caption writing.
- Quality: 98.0% of 1,582 checked nouns and verbs were semantically accurate, while transcription errors were 3.3% missing, 2.2% extra, and 1.1% typographical words.Experts independently checked every word in 100 randomly selected COCO annotations.
- Localization quality: Trace points outside object boxes commonly arise from circling objects or from annotators moving before or after describing them.The authors frame these timing effects as opportunities to better understand vision-language connections.
- Richness: Localized Narratives captions average 36.5 words and contain more nouns, pronouns, adjectives, verbs, and adpositions per caption than most previous datasets.The captions are also substantially longer than previous datasets except Stanford Visual Paragraphs.
- Diversity: The dataset has wider noun-count variation than several prior datasets and supports diverse trace styles including circling, scribbling, and underlining.Annotators were allowed to hover over objects in ways they considered natural rather than following a bounding-box format.
5 Controlled Image Captioning
Controlled image captioning uses mouse traces to specify which image regions to describe and in what order, producing captions better aligned with user intent than standard captioning.
- Task definition: The task takes an image and mouse trace as input, producing a caption that describes traced regions in trace order.The trace provides both spatial selection and ordering information.
- Method: The modified captioning model adds mouse-trace position and duration to visual proposal features during training and testing.The input representation includes semantic and location features for detected proposals, total trace duration, and trace position.
- Qualitative results: Qualitative examples show that traces focus attention on otherwise missed objects and produce more complete, fine-grained descriptions.Different traces can produce different captions, while trace order maps to caption word order.
- Quantitative results: ROUGE-1-F1 increased from 0.479 for standard captioning to 0.607 with the full mouse trace.Because ROUGE-1-F1 ignores word order, the increase reflects caption completeness.
- Quantitative results: Controlled captioning achieved 0.483 ROUGE-L versus 0.317 for standard captioning, a 0.166 improvement.The smaller relative drop from ROUGE-1-F1 to ROUGE-L indicates that the model exploits trace order.
- Interpretation: The authors conclude that mouse-trace captions fit the user's requested image regions and ordering better, without claiming superior absolute caption quality.Ablations indicate that most improvements come from the trace's location and order information.
6 Conclusions
The paper introduces Localized Narratives, scales them to 849k images, analyzes their grounding, and demonstrates their utility for controlled image captioning.
- Contributions: Localized Narratives visually ground every word of an image caption with a mouse trace.The annotations connect image, spoken description, textual description, and mouse trace modalities.
- Contributions: The dataset contains Localized Narratives for 849k images and supports analysis of rich, accurate grounding.The annotations are also presented as an efficient way to collect image captions.
- Applications: The paper demonstrates the data's utility through controlled image captioning using mouse traces.
Appendices
The appendices provide a demonstration of image generation, additional controlled-captioning examples, localization analysis, and framework details.
- Appendix A: Appendix A demonstrates an image-generation application using Localized Narratives.
- Appendix B: Appendix B provides additional qualitative examples for controlled image captioning.
- Appendix C: Appendix C adds a quantitative plot of mouse-trace localization accuracy for Open Images.The plot was omitted from the main paper because of space limitations.
- Appendix D: Appendix D gives additional technical details about the controlled image captioning framework.
A Image Generation
Localized Narratives provide an intuitive interface for image generation by letting users specify image elements through labeled mouse traces, which are converted into segmentation maps for SPADE.
- Motivation: The image-generation application uses labeled mouse traces as a natural alternative to directly providing semantic segmentation maps.The interface lets users specify both the location and class label of desired elements.
- Framework: The method starts from SPADE, a framework that generates images conditioned on pixel-wise semantic segmentation maps.The model is pretrained on COCO-Stuff with 182 object and background classes.
- Map construction: Retrieved masks are iteratively added to an initially unlabeled segmentation map in trace-segment order.Object masks overwrite prior labels, while background masks overwrite only other background labels.
- Results: Generated examples become increasingly complex as the narrative continues while preserving previous details.Adding indicated elements can alter existing scene properties, such as boat visibility, water appearance, and weather.
- Conclusion: The authors demonstrate incremental generation with an intuitive interface rather than claiming better image quality.The framework keeps the pretrained SPADE model unmodified and currently generates nouns only.
B Additional Qualitative Examples for Controlled Image Captioning
Figures 10 and 11 provide additional qualitative comparisons between classical image captioning and captioning controlled by mouse traces. Figure 9 illustrates iterative image generation by adding elements from nouns and associated trace segments.
- Image Generation: Figure 9 shows seven image-generation examples in which new elements are iteratively added from left to right.Each added element uses a noun together with its associated trace segment.
- Qualitative Examples: The example descriptions mention snow-covered ground, a skier, clothing, and background plants.These examples include a person skiing with a ski board and wearing gloves, goggles, a helmet, and a red jacket.
- Controlled Captioning: Figures 10 and 11 compare traditional captioning using only the image with mouse-trace-controlled captioning.In the controlled setting, mouse traces are provided as an additional model input.
- Controlled Captioning: The controlled-captioning figures use a gradient to indicate time in the mouse traces.This temporal encoding appears in both qualitative-example figures.
C Localization accuracy on Open Images
Figure 12 presents mouse-trace segment locations on COCO and Open Images relative to the closest bounding box of the relevant class.
- Localization Accuracy: Figure 12 shows histograms of mouse-trace segment locations for COCO and Open Images.COCO appears on the left and Open Images on the right.
- Localization Accuracy: The histograms measure trace-segment locations with respect to the closest box of the relevant class.The figure uses the closest class-relevant bounding box as the spatial reference.
D.1 Method and Training Details
The captioning model follows a transformer encoder-decoder architecture with reduced depth, normalized projection layers, fewer training iterations, and support for longer target captions.
- Architecture: The transformer encoder and decoder each use 2 layers instead of 6.This is one of the stated architectural differences from the referenced work.
- Architecture: The projection layers include layer normalization.The paper identifies this as another difference from the referenced architecture.
- Training: Training uses a maximum of 150k iterations rather than 2M.The stated iteration limit is much smaller than that used in the referenced work.
- Training: The model permits target captions of up to 225 tokens.This maximum length is used to account for longer descriptions.
- Input Features: Input features are standard regional Faster R-CNN features without ultra-finegrained, global, or entity features.Additional feature representations are described separately in Section D.2.
D.2 Representations of visual and trace features
The model combines regional visual features, object locations, trace duration, and trace positions, while preserving trace order through temporal representations and concatenating visual and trace sequences.
- Feature Types: The model can consume Faster R-CNN object features, proposal coordinates and sizes, total trace duration, and mouse-trace positions.These features represent object semantics, detected-object locations, description length, and visual grounding.
- Trace Features: Mouse traces are divided into 0.4-second pseudo-segments and represented by bounding boxes.Each pseudo-segment produces a 5D vector with the same form as the detected-object location features.
- Visual Features: Visual object features use 16 projected 2,048D Faster R-CNN vectors and corresponding 5D location vectors.The feature vectors are projected to 512D and followed by layer normalization.
- Trace Features: Trace boxes can be expanded by offset δ to control whether position information is retained.δ = 1.0 removes trace-position information and retains total duration, whereas δ = 0.1 provides duration and position signals.
- Temporal and Sequence Encoding: Trace order is encoded with 512D sinusoidal time vectors, while visual features omit time encoding to remain permutation-invariant.The resulting visual and trace feature sequences are concatenated as the model input.