PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, Fei-Fei Li

arXiv:1602.07332v1cs.CVcs.AI

TL;DR

Computers lack datasets that capture the objects, attributes, and relationships needed for detailed visual understanding. Visual Genome addresses this gap with dense, localized annotations and question-answer pairs, providing a structured representation of images for vision research.

  • Problem

    Existing image datasets emphasize object recognition and simple descriptions, while detailed understanding of object interactions remains limited despite its importance for scene understanding and question answering.

  • Method

    Visual Genome collects dense, region-localized descriptions, object attributes, relationships, and freeform and region-based question-answer pairs across images.

  • Results

    The dataset provides a structured, formalized representation of images that links visual regions with textual objects, attributes, relationships, and questions.

  • Takeaways & Limitations

    Visual Genome offers a comprehensive dataset for training and benchmarking models across object recognition, relationship understanding, and visual question answering.

  • Takeaways & Limitations

    Relationship prediction can capture salient scene features but may fail to distinguish between visually similar objects, such as boys versus women or cars versus buses.

Abstract

from arXiv · show

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) in order to answer correctly that "the person is riding a horse-drawn carriage". In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 100K images where each image has an average of 21 objects, 18 attributes, and 18 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answers.

1 Introduction

Visual Genome targets comprehensive scene understanding by linking visual concepts to language and representing objects, attributes, and relationships within images. It provides dense, region-based descriptions and question-answer annotations to support cognitive vision tasks such as image description and question answering.

  • Motivation: Complete scene understanding requires detecting objects, describing their attributes, and recognizing relationships and interactions among them.These capabilities support applications including image search, question answering, and robotic interaction.
  • Dataset goals: Visual Genome addresses three dataset gaps: grounding visual concepts to language, collecting region-specific descriptions and QAs, and formalizing image components.The dataset is designed for training and benchmarking comprehensive scene-understanding models.
  • Annotation scope: Relationships and attributes are first-class annotations because reversing a relationship can change a scene’s meaning, such as dog chasing a man versus man chasing a dog.The dataset explicitly extends beyond the traditional focus on object recognition.
  • Dense descriptions: More than 42 descriptions are collected for different image regions, capturing multiple stories that a single summary sentence cannot fully express.This region-based approach contrasts with datasets focused primarily on high-level overall descriptions.
  • Question answering: An average of 17 question-answer pairs is collected per image from the descriptions, enabling joint development of language and vision models.The region-based QAs can support answering questions from the description, the image, or both.
  • Structured representation: Dense descriptions and explicit links between image bounding boxes and textual descriptors support a structured, formalized image representation akin to knowledge-base representations.The representation connects visual pixels with relationships and attributes in a scene.

2 Visual Genome Data Representation

Visual Genome represents images through seven complementary components, beginning with unconstrained human descriptions and question-answer pairs and extracting structured objects, attributes, relationships, and graph representations. Its annotations localize content to regions, connect objects through directed canonicalized relations, and support both whole-image and region-based questions.

  • Dataset components: Seven components organize the dataset: region descriptions, objects, attributes, relationships, region graphs, scene graphs, and question-answer pairs.Descriptions and question answers are collected as raw text before structured annotations are extracted.
  • Region descriptions: Human-generated descriptions cover multiple overlapping image regions, each localized by a bounding box.This region-based design addresses the insufficiency of a single summary sentence for representing an image’s contents and interactions.
  • Objects and attributes: 21 objects per image on average are delineated by tight bounding boxes and canonicalized to WordNet synset IDs.Canonicalization also connects synonymous object names and related concepts across images.
  • Objects and attributes: 16 attributes per image on average describe object properties or states, with attributes canonicalized to WordNet.Examples include color and state annotations extracted from region descriptions.
  • Relationships and graphs: 18 relationships per image on average connect objects through directed actions, spatial relations, verbs, prepositions, and comparisons.Each relation runs from a subject object to an object and is canonicalized to a WordNet synset ID.
  • Question-answer pairs: Freeform and region-based QA pairs cover what, where, how, when, who, and why questions, with at least one of each type per image.Freeform questions concern the entire image, while region-based questions use selected regions and their descriptions.

3 Related Work

Prior vision datasets largely targeted individual perceptual tasks such as classification, detection, segmentation, and captioning, while Visual Genome was designed as a general-purpose representation combining dense objects, attributes, relationships, descriptions, and question-answer pairs. Related work also shows the importance of scene-graph relationships and exposes limitations in existing image-description and visual-question-answering datasets.

  • Datasets: Earlier datasets progressed from hand-curated classification categories to larger, crowdsourced datasets supporting object detection, segmentation, and related vision benchmarks.Caltech 101 contained 101 categories, Caltech 256 expanded category coverage, Pascal VOC contained 20 categories in 11,000 images, and ImageNet contained 14 million images.
  • Datasets: Visual Genome bridges these datasets by collecting objects, scene graphs, region descriptions, and question-answer pairs as a task-independent representation of the visual world.Its images contain an average of 21 objects per image.
  • Datasets: Visual Genome provides an order of magnitude more unique objects, attributes, and relationships than other datasets, along with 1.7 million visual question-answer pairs.The dataset comparison also reports more descriptions and question answers, more diverse object, attribute, and relationship classes, and higher annotation density per image.
  • Image Description and Visual QA: Existing captioning datasets often describe only salient image content, while visual QA benchmarks concentrate on simple recognition questions with short answers.Ninety percent of DAQUAR answers and 87% of VQA answers consist of single-word object names, attributes, or quantities.
  • Relationships: Scene-graph research represents objects and attributes as nodes and relationships as edges, supporting image-to-sentence mapping, image generation from sentences, and image search.The passage identifies scene graphs as structured representations in which relationships connect object nodes.

4 Crowdsourcing Strategies

Visual Genome used a multi-stage Amazon Mechanical Turk pipeline to collect and verify dense annotations, emphasizing accuracy, coverage, and diversity. The process iteratively built region descriptions, grounded objects, attributes, relationships, scene graphs, and question-answer pairs.

  • Crowdsourcing Pipeline: Over 33,000 unique workers completed approximately 800,000 AMT HITs during six months of dataset collection.The collection followed 15 months of experimentation and iteration on the data representation.
  • Region Descriptions: Each image received 50 region descriptions through repeated rounds that encouraged workers to describe previously unannotated regions.BLEU thresholds of 0.7 filtered descriptions similar to image-specific and globally common descriptions, while bounding boxes were required to cover all mentioned objects.
  • Objects and Co-reference: Object extraction grounded nouns in bounding boxes while suggested prior annotations enabled co-reference chains across descriptions.A dependency parser supplied noun suggestions, and combining parsing with crowdsourcing accelerated extraction without losing accuracy.
  • Scene Graphs: Scene graphs merged region-graph nodes when bounding-box intersection-over-union exceeded 0.8, then asked workers to confirm proposed merges.This worker verification step addressed false positives from the overlap heuristic.
  • Question-Answer Pairs: Question-answer collection constrained workers to write seven-W questions that were unambiguous, precise, unique, and answerable from the image.The rules covered who, what, where, when, why, how, and which questions.

5 Dataset Statistics and Analysis · 5.1 Image Selection

Visual Genome analyzes statistics across images, region descriptions, questions and answers, and graph components. Its 108,249 real-world images span 972 WordNet synsets but are somewhat biased toward people and sports scenes.

  • 5 Dataset Statistics and Analysis: The analysis covers image distributions, region descriptions, questions and answers, and region and scene graphs.Objects, attributes, and relationships are also analyzed as constituent graph components.
  • 5.1 Image Selection: 108,249 images were selected from the intersection of MS-COCO’s 328,000 images and YFCC’s 100 million images.The images are real-world, non-iconic photographs uploaded to Flickr by users.
  • 5.1 Image Selection: Image widths range from 72 to 1280 pixels, with an average width of 500 pixels.These dimensions characterize the selected Visual Genome image collection.
  • 5 Dataset Statistics and Analysis: Region descriptions average 5 words, ranging from 1 to 16 words.This summarizes the reported distribution of region-description lengths.
  • 5.1 Image Selection: The image distribution is somewhat biased toward people, with the three most common synsets referring to people playing sports.The cited examples are “ski,” “ballplayer,” and “racket.”

5.2 Region Description Statistics

Visual Genome provides dense, diverse region descriptions that span specific objects and broader image concepts. Their statistics show variation in region scale, description length, visual vocabulary, and semantic content across images.

  • Region coverage: 42 regions per image are annotated with bounding boxes and descriptive phrases spanning single objects, multiple objects, and salient image content.Examples range from “A bag” to “Man taking a photo of the elephants.”
  • Description diversity: Visual Genome descriptions emphasize different image regions and are therefore less semantically similar than multiple scene-level captions of the same image.Description diversity is additionally controlled using BLEU-score thresholds against previously written descriptions.
  • Region coverage: Region scale controls description granularity: small regions tend to yield specific object descriptions, whereas large regions tend to express general image-level concepts.The paper illustrates this contrast with “A bag” and “An man being chased by a bear.”
  • Vocabulary: Colors occur most frequently in the description vocabulary, followed by common objects such as “man,” “tree,” and “sign,” and universal elements such as “sky.”The dataset also contains frequent visual phrases such as “blue sky” and “green grass.”
  • Semantic diversity: Descriptions are converted into 300-dimensional word2vec representations after stop-word removal and averaging, enabling unsupervised semantic clustering.The pipeline applies hierarchical agglomerative clustering to the resulting vectors.
  • Semantic diversity: 2 clusters are represented on average in each image’s MS-COCO captions, providing a baseline for comparing per-image semantic diversity.The comparison aggregates Visual Genome and MS-COCO descriptions, removes stop words, and measures cluster diversity per image.

5.3 Object Statistics

Visual Genome combines high object density with unusually broad category diversity relative to related datasets. Its images average around 21 unique objects, while object frequencies reflect common visual subjects such as people, animals, sports, and scenery.

  • Object density and diversity: Approximately 21 objects per image places Visual Genome ahead of ImageNet, PASCAL, MS-COCO, and other related datasets in object density.The passage describes these margins as large, though it does not provide the comparison values for each dataset.
  • Object density and diversity: Visual Genome represents more object categories than any other dataset in the comparison.The dataset’s object diversity is especially relevant when comparing against MS-COCO.
  • Object categories: Humans, animals, sports, and scenery are among the most common WordNet-related object categories, with man, person, and woman occurring especially frequently.This distribution is consistent with the dataset’s general bias toward these image subjects.
  • Object distribution: Between 0 and 2 objects occur in each region on average, including zero when a region description refers to no explicit object.Regions containing one object generally focus their descriptions on that object.
  • Object distribution: Around 21 unique objects occur in each image on average, while few images contain fewer objects or more than 40.The distribution is concentrated away from both very low and extremely high object counts.

5.4 Attribute Statistics

Visual Genome’s attribute annotations provide detailed object descriptions, with broad coverage and frequent co-occurrence patterns. Attribute cliques can characterize object types based solely on their associated traits.

  • Attribute coverage: About 45% of objects have at least one attribute, totaling 1.6 million attributes across 13,041 unique attributes.Attributes include colors, sizes, action verbs, and materials; each attribute belongs to one object, while objects may have multiple attributes.
  • Attribute coverage: 21 attributes occur per image on average, while each region averages 1 attribute and 42% of regions contain none.The absence of attributes is primarily attributed to relationship-focused regions.
  • Attribute distributions: Colors are the most frequent attributes, followed by sizes and materials.Common examples include white, green, large, and wooden.
  • Attribute graphs: Attribute co-occurrence graphs connect attributes describing the same object, and cliques capture groups with pairwise co-occurrence.These cliques provide insight into commonly perceived traits associated with different object types.
  • Attribute graphs: Attribute cliques can identify object types, including animals, water bodies, human hair, skateboarders, and subsets of cars.The skateboarder clique includes athletic, young, fit, skateboarding, focused, teenager, male, skinny, and happy; a car clique includes shiny, small, metal, silver, rusty, parked, and empty.

5.5 Relationship Statistics

Visual Genome encodes directional relationships between subject–object pairs, spanning spatial, action, and compositional interactions. The dataset averages one relationship per region and 19 per image, with frequent relationships revealing spatial patterns, human actions, and object affordances.

  • Relationship representation: Relationships are directional subject–object predicates that capture spatial, action, and compositional interactions between objects.They are written as relationship(subject, object), such as swinging(man, bat).
  • Relationship density: 1 relationship per region on average, with a maximum of 7; each image contains 19 relationships on average.Some descriptions contain multiple attributes but no relationships, while the number of relationships per image has a minimum of 1.
  • Relationship distributions: on is the most frequent relationship, while holding, wearing, behind, next to, and under are also common.The word on is especially frequent because it covers spatial configuration and attachment; human-involving relationships are more action-oriented.
  • Understanding affordances: Riding relationships reveal affordance patterns in which people-like entities commonly ride other objects, while skateboarders ride skateboards and surfers ride waves or surfboards.The subject and object distributions for riding expose recurring common-sense associations.

5.6 Region and Scene Graph Statistics

Visual Genome introduces the largest scene-graph dataset to date, representing images through fine-grained region graphs aggregated into one scene graph per image. The dataset averages 42 region graphs per image and scene graphs with 21.26 objects, 16.21 attributes, and 18.67 relationships.

  • Overview: The paper presents Visual Genome as the largest scene-graph dataset to date for deeper representations of the visual world.The section analyzes these representations at both region and image levels through region graphs and scene graphs.
  • Region and scene graph construction: Visual Genome collects scene graphs at a finer-grained region level than prior human-written image triples.Workers create region graphs from collected region descriptions, producing multiple graphs for each image.
  • Region and scene graph construction: Region graphs from each image’s region descriptions are aggregated into one scene graph, merging multiply referenced objects into single nodes.Each scene graph contains the individual region graphs after object merging.
  • Dataset statistics: 21.26 objects, 16.21 attributes, and 18.67 relationships are the average counts per scene graph and image.Region graphs average 0.43 objects, 0.41 attributes, and 0.45 relationships, with these quantities approximately normally distributed.

5.7 Question Answering Statistics

Visual Genome contains 1,773,258 question-answer pairs designed to span six W-based question categories, from perceptual recognition to complex common-sense reasoning. The dataset emphasizes diverse, non-binary questions with varied answer lengths, enabling stratified analysis of question-answering difficulty.

  • Dataset scale and organization: 1,773,258 question-answer pairs cover Visual Genome images, averaging 17 pairs per image and beginning with one of the six Ws.The pairs address each image’s content through a question and its correct answer.
  • Question categories: Six W categories range from basic perception to common-sense reasoning and provide a consistent difficulty stratification.Why questions involving complex reasoning achieve the lowest baseline performance, at 3.4% top-100 accuracy.
  • Question type distributions: “What” questions are the largest category, comprising nearly half of the question-answer pairs.Questions are categorized by their interrogative words and then refined using subsequent words.
  • Question type distributions: Binary questions are ruled out to encourage more difficult question-answer pairs rather than allowing trivial yes/no prediction.In VQA, 32.37% of questions are yes/no binary questions, which can support reasonable performance from predicting “yes” or “no.”
  • Question and answer length distributions: The average question and answer lengths are 5.7 and 1.8 words, while answer lengths follow a long-tail distribution.Answers contain one, two, or three words in 57.3%, 18.1%, and 15.7% of cases, respectively.

5.8 Canonicalization Statistics

Visual Genome canonicalizes objects, attributes, and relationships by mapping them to WordNet synsets, reducing conceptual noise and connecting the dataset to research resources. The process covers region descriptions and QA pairs, using extraction, normalization, and sense-mapping procedures with reported accuracy and recall measurements.

  • Canonicalization approach: Canonicalization applies word sense disambiguation by mapping Visual Genome objects, relationships, and attributes to WordNet synsets.This mapping is intended to reduce noise in dataset concepts and facilitate connections to other research resources.
  • Region descriptions and QAs: All objects mentioned in region descriptions and question-answer pairs are canonicalized after noun-phrase extraction with Stanford NLP tools.The extraction procedure achieved 99% noun-phrase recall on a manually annotated subset of 200 region descriptions.
  • Region descriptions and QAs: 86% overall mapping accuracy and 98.5% recall were obtained by assigning extracted noun phrases to their most frequent matching WordNet synsets.The frequency-based mapping uses WordNet lexeme counts to select synsets.
  • Attributes: Attributes are normalized morphologically and mapped to frequent WordNet synsets, with 15 hand-mapped rules addressing common spatial-sense failures.The procedure avoids relying on part-of-speech taggers because scene-graph attributes are disjoint elements.
  • Attributes: 95.9% recall and 83.5% mapping accuracy were achieved when algorithmic attribute mappings were compared with hand-produced ground truth from 200 sampled attributes.The evaluation addresses cases where WordNet frequency counts favor abstract rather than visually appropriate senses.
  • Relationships: Relationships are canonicalized after excluding prepositions and using verb morphology and syntactic placement to select matching WordNet sentence frames.The method primarily handles verb relationships and considers synsets with matching sentence frames.

6 Experiments

The experiments establish baseline results for Visual Genome’s underexplored attributes, relationships, region descriptions, and visual question-answer components. Jointly modeling objects with attributes or relationships improves prediction, while region captioning outperforms Flickr30K and question answering reveals a long-tailed answer distribution.

  • Experimental scope: The experiments focus on attributes, relationships, region descriptions, and question-answer pairs rather than established object detection and image-retrieval tasks.The section presents baseline results for components not extensively studied.
  • Attribute prediction: 18.97% top-one and 43.11% top-five accuracy were achieved for independent attribute prediction, versus 43.17% and 71.97% when objects and attributes were predicted jointly.Joint learning increases inter-class variance and exploits attributes associated with limited object classes.
  • Relationship prediction: 8.74% top-one and 29.69% top-five accuracy were obtained for relationship classification, increasing to 25.83% and 65.57% when the relationship and both object classes were predicted jointly.Relationships were difficult to learn because they were not visually distinct enough and had high intra-class variability; object classification aided prediction.
  • Region captioning: The Visual Genome-trained region-captioning model outperformed the Flickr30K-trained model on BLEU, CIDEr, and METEOR, with crowd-evaluated correctness of 43.03% versus 1.6%.The two models generated different types of descriptions, so region-captioning results are not directly comparable to image-captioning results.
  • Visual question answering: Visual Genome contains 1.7 million question-answer pairs, averaging 17 pairs across each of 108,249 images, and provides baseline metrics for visual question answering.The task requires capabilities including fine-grained recognition, object detection, and activity recognition.
  • Visual question answering: The top 100, 500, and 1000 most frequent answers cover only 41.1%, 57.3%, and 64.1% of correct answers, respectively, indicating a long-tail answer distribution.The corresponding VQA coverage is higher at 63%, 75%, and 80%.

7 Future Applications

Visual Genome enables applications beyond its baseline experiments, including dense scene description, question answering, image-understanding evaluation, visual relationship extraction, and semantic image retrieval. These applications exploit its captions, question answers, scene graphs, region descriptions, and region graphs to model or evaluate image content more comprehensively.

  • Dense image captioning: Dense captioning models can describe individual parts of a scene rather than exhaustively representing an entire image with one caption.The dataset enables this extension of conventional image captioning by supporting descriptions of scene parts.
  • Visual question answering: Visual Genome combines question answers with descriptions and scene graphs, enabling supervised models that use these components for visual question answering.Future systems can integrate multiple dataset components rather than treating question answering as an isolated task.
  • Image understanding: Visual Genome scene graphs can support more comprehensive image-understanding evaluation than BLEU, CIDEr, or METEOR, which do not effectively measure model understanding.The passage identifies a need for evaluation metrics beyond commonly used captioning and question-answering measures.
  • Relationship extraction: Visual Genome provides a large-scale visual relationship dataset for extracting relationships from images and studying action recognition through object interactions.The dataset is described as the first large-scale visual relationship dataset.
  • Semantic image retrieval: Semantic image retrieval can combine region descriptions with region graphs, while attention-based search can localize a query-specified area in retrieved images.These methods extend prior scene-graph approaches to semantic search by incorporating localized regions of interest.

8 Conclusion

Visual Genome provides a multi-layered, comprehensive dataset for studying images from pixel-level objects through inferred relationships to cognitive tasks such as question answering. It is intended to train and benchmark next-generation computer vision models toward broader visual-world understanding.

  • Conclusion: Visual Genome supports multi-perspective image analysis spanning objects, relationships requiring inference, and cognitive tasks such as question answering.The dataset links pixel-level information to increasingly inferential and cognitive forms of image understanding.
  • Conclusion: Visual Genome is a comprehensive dataset for training and benchmarking the next generation of computer vision models.The conclusion presents the dataset as a resource for both model development and evaluation.
  • Conclusion: The authors expect Visual Genome to help models develop a broader understanding of the visual world.This goal is framed as complementing computers’ existing capabilities.
Loading 1602.07332v1…