PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

Xiao Lin, Devi Parikh

arXiv:1502.06108v3cs.CV

TL;DR

Text-based common-sense resources may miss unwritten knowledge, while systems struggle with tasks requiring common-sense reasoning. This paper imagines abstract scenes and combines visual with textual cues, outperforming a strong text-only baseline on fill-in-the-blank and visual paraphrasing.

  • Problem

    Most common-sense knowledge bases rely on text, although much common-sense knowledge goes unmentioned and systems struggle to answer questions requiring such reasoning.

    Source: §1 · p. 1

  • Method

    The approach imagines abstract scenes behind text and reasons jointly over visual and textual common sense to solve fill-in-the-blank and visual paraphrasing tasks.

    Source: §1 · p. 2

  • Results

    The approach outperformed the text-only baseline on both tasks; on FITB, it achieved 48.04% accuracy versus 44.97%, a 3.07% gain.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Imagined scenes can serve as intermediate representations for encoding visual common sense in non-visual tasks.

    Source: §6 · p. 8

Abstract

from arXiv · show

Artificial agents today can answer factual questions. But they fall short on questions that require common sense reasoning. Perhaps this is because most existing common sense databases rely on text to learn and represent knowledge. But much of common sense knowledge is unwritten - partly because it tends not to be interesting enough to talk about, and partly because some common sense is unnatural to articulate in text. While unwritten, it is not unseen. In this paper we leverage semantic common sense knowledge learned from images - i.e. visual common sense - in two textual tasks: fill-in-the-blank and visual paraphrasing. We propose to "imagine" the scene behind the text, and leverage visual cues from the "imagined" scenes in addition to textual cues while answering these questions. We imagine the scenes as a visual abstraction. Our approach outperforms a strong text-only baseline on these tasks. Our proposed tasks can serve as benchmarks to quantitatively evaluate progress in solving tasks that go "beyond recognition". Our code and datasets are publicly available.

1. Introduction

The paper introduces fill-in-the-blank (FITB) and visual paraphrasing (VP) as tasks that benefit from visual common sense, and an imagination-based approach that outperforms a text-only baseline on both. It frames the tasks as benchmarks for image understanding beyond recognition.

1. Introduction

  • Although AI agents can answer factual questions, common sense continues to elude them, and current systems cannot reliably answer the paper’s example FITB and VP questions.The examples require reasoning about dangerous bears, hiding to avoid notice, and people throwing things when angry.Source: §1 · p. 1
  • Text-based common-sense resources are limited because people rarely state ordinary knowledge, text favors unusual or foreground details, and some useful knowledge is unnatural to articulate.For example, describing someone running after another person normally leaves implicit their facing, gaze, and running poses.Source: §1 · pp. 1–2
  • Visual common sense is semantic knowledge learnable from visual data, such as what a person running after someone would see if that person turned around, and complements non-visual sources.The paper distinguishes this knowledge from models of common object interactions or visual relationships, parts, and attributes.Source: §1 · p. 2
  • FITB and VP can benchmark image understanding beyond recognition by testing language grounding and common-sense learning from visual data.The tasks are presented as textual on the surface but can benefit from visual common sense.Source: §1 · p. 2
  • The approach imagines an abstract scene for each candidate description, then jointly reasons over text and vision to select an answer or classify whether two descriptions depict the same scene.For FITB, it generates scenes for four description-option pairs; for VP, it generates a scene for each of two descriptions.Source: §1 · p. 2
  • The imagination-based approach, which leverages visual and textual common sense, outperforms a text-only baseline on both FITB and VP.The authors introduce datasets for both tasks and make the code and datasets publicly available.Source: §1 · p. 2

2. Related Work

Prior beyond-recognition tasks test higher-level image understanding but can be difficult to evaluate; this work presents FITB and VP as multiple-choice benchmarks that are easier to score.

Beyond recognition

  • Image descriptions and text-to-scene synthesis broadly test higher-level understanding, but are hard to evaluate and may reward grammatical correctness.Beyond-recognition tasks also include reasoning about what people discuss, when events occur, where images were taken, and why people act.Source: §2 · p. 2
  • Unlike factual natural-language QA, FITB asks what is plausible in a partial scene and VP how descriptions can vary; both are multiple-choice and easy to evaluate.Both require a level of image understanding similar to image-to-text and text-to-image tasks.Source: §2 · p. 2

Leveraging common sense

  • Prior work shows non-visual common sense benefits physical reasoning, intention reasoning, and object functionality; this work explores visual common sense for non-visual tasks by generating scenes.Visual context has also been used to improve recognition.Source: §2 · p. 3

Synthetic data

  • Human-created clipart trains the models and supplies imagined scenes for FITB/VP, providing semantically rich variations without relying on imperfect object recognition or detection.Although abstract scenes are not photorealistic, semantic concepts learned from them have been shown to generalize to real images.Source: §2 · p. 3

3. Dataset

The paper builds fill-in-the-blank and visual-paraphrasing datasets from Abstract Scenes to evaluate reasoning about scenes and descriptions; human results indicate that both tasks are solvable, though FITB includes substantial ambiguity.

3. Dataset

  • The datasets use 10,020 Abstract Scenes images depicting a boy and girl in a park, represented with 58 clipart objects and paired descriptions.Each scene has two descriptions written by different people; the scenes also provide the clipart representation used to imagine scenes and learn visual common sense.Source: §3 · p. 3

3.1. Fill-in-the-blank (FITB) Dataset

  • Across FITB question subsets defined by minimum human-agreement levels, mode-response accuracy (mode matching ground truth) correlated well with human agreement; even random responses yield 43% average agreement with the mode.Source: §3.1 · p. 4
  • FITB questions omit one of three source-description sentences, then ask readers to select it from that sentence and three distractors drawn from other scenes.After degenerate parser outputs were removed, the dataset contained 8,959 questions: 7,198 for training and 1,761 for testing; test scenes are unavailable at test time.Source: §3.1 · p. 3

3.2. Visual Paraphrasing (VP) Dataset

  • VP asks whether two descriptions refer to the same scene, using paired descriptions of one scene as positive examples and descriptions from different scenes as negatives.The dataset has 30,060 pairs—10,020 positive and 20,040 negative—with 24,000 for training and 6,060 for testing; negative pairs do not overlap across splits.Source: §3.2 · p. 4
  • Human responses achieved 94.78% average precision on VP, compared with 33% chance.Ten subjects’ yes/no responses were averaged into a score for each question to compute the precision-recall curve.Source: §3.2 · p. 4
  • The tasks are proposed as benchmarks for visual common sense, with possible applications in story generation, question answering, image retrieval, and diverse image-description generation.FITB may support automatic storytelling and question answering; VP may support text-based image retrieval and generating multiple descriptions of one image.Source: §3.2 · p. 4

4. Approach

The approach extends a text-only baseline by imagining scenes and scoring their visual plausibility and consistency with descriptions. Generated-scene features are then used to answer fill-in-the-blank and visual-paraphrasing questions.

4.1. Text Only Model

  • The text-only baseline represents description pairs with term-frequency, word co-occurrence, and averaged word2vec features, then uses a ranking SVM for FITB and a binary SVM for VP.The VP representation combines features from both description orders with their absolute difference to ensure symmetry.Source: §4.1 · pp. 4–5

4.2.1 Scene and Description Encoding

  • Scene and description encoding represents objects by presence, location, depth, orientation, and attributes, while descriptions become semantic-role tuples mapped to 58 objects.The datasets contain 1,133 nouns and 2,379 relations; tuple extraction uses semantic role analysis.Source: §4.2.1 · p. 5

4.2.2 Visual Common Sense

  • Visual common sense is modeled with shared unary factors for object states and pairwise factors for object co-occurrence and relative properties, represented by log-likelihood features learned for FITB or VP.The location distributions are modeled with Gaussian mixture models because the location space is large.Source: §4.2.2 · pp. 5–6

4.2.3 Text-Image Consistency

  • Text-image consistency scores mentioned objects and relations using noun-conditioned presence and attributes plus relation-conditioned relative properties, with default factors for unmentioned objects and pairs.Relation factors are shared across objects so the same relation can have similar visual instantiations across different object pairs.Source: §4.2.3 · p. 6

4.3. Scene Generation

  • Scene generation extracts tuples from a description and uses Zitnick et al. ’s CRF approach to sample scenes with Iterative Conditional Modes from different initializations.The CRF scoring function is similar to the model’s visual-plausibility and text-image-consistency terms.Source: §4.3 · pp. 6–7

4.4. Answering Questions with Imagined Scenes

  • For FITB, the model generates a scene for each question–answer pair and uses scoring-function features to train a ranking SVM; for VP, it scores each generated scene against the other description and trains a symmetric-feature binary SVM.The SVM C parameter is selected using 5-fold cross-validation for both tasks.Source: §4.4 · p. 7

5. Experiments and Results

Imagined visual-scene reasoning improves performance on the text-input FITB and visual paraphrasing tasks. The experiments also show that visual-feature contributions and the value of inferring unmentioned objects differ by task.

5.1. Fill-in-the-blank

  • On FITB, imagined visual-text reasoning reaches 48.04% accuracy, beating the text-only baseline’s 44.97% by 3.07% with 22 extra feature dimensions, near human performance at 52.87%.The baseline uses 2,486 dimensions.Source: §5.1 · p. 7, Table 1 · p. 7
  • On FITB, presence and attribute features score 47.02% and 46.39%, spatial information scores 44.80%, and presence plus attribute reaches 48.60%, above all three features together.The all-three result is 48.04%.Source: §5.1 · p. 7
  • FITB text-only and vision-plus-text systems make complementary errors: at least one answers 54.68% of questions correctly, rising to 75.92% when human responses are included.Source: §5.1 · p. 7
  • On FITB human-mode responses, the approach matches the majority vote 39.35% of the time versus 36.40% for text alone; retraining on mode labels yields 45.43% versus 42.25%.The authors note that this task is harder than predicting ground truth, where the text-only baseline scores 44.97%.Source: §5.1 · p. 7
  • On FITB, the visual approach’s margin over the baseline grows from 3% across all questions to 6% on questions with high human agreement.The evaluation compares subsets by the number of people agreeing with the human-mode response.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2. Visual Paraphrasing

  • On visual paraphrasing, scene generation and reasoning outperform text-only reasoning by 1.4% and push performance above human performance, though the paper attributes this to possible MTurk noise.Source: §5.2 · p. 8, Table 2 · p. 8
  • On visual paraphrasing, presence contributes 0.93% and spatial configuration 0.60%; imagining mentioned objects alone scores 95.01%, close to 95.08% with extra objects inferred.The paper suggests unmentioned objects matter more in FITB, while visual paraphrasing depends on attributes and spatial configurations because relevant objects tend to be stated.Source: §5.2 · p. 8

6. Discussion

The discussion frames imagined clipart scenes as intermediate representations for visual common sense and outlines ways to improve scene-based reasoning and extend it to other tasks.

  • Imagined scenes serve as intermediate representations that encode visual common sense, much as parts and attributes support recognition [42].Source: §6 · p. 8
  • Clipart scenes offer a non-explicit way to imagine scenes, unlike knowledge bases that explicitly represent instance relations, and resemble physical simulations.The authors note that humans cannot always verbalize their reasoning and that more explicit representations of visual common sense remain worth investigating.Source: §6 · p. 8
  • A scene-generation model that infers objects beyond those mentioned in text performs better in the experiments.Source: §6 · p. 8
  • Improved scene generation, answer features, and modalities could help; proposed directions include multiple diverse images, richer joint text-image representations, and jointly learning scene generation and visual common-sense models with CRF-like scoring.Source: §6 · p. 8
  • The authors propose studying whether other tasks could benefit from this intermediate representation.Source: §6 · p. 9

Non-Visual Tasks

The section evaluates visual paraphrasing at coarse and fine granularity and reports that visual features improve on a text-only baseline in both settings.

Coarse and Fine-grained Visual Paraphrasing

  • The 10,020 Abstract Scenes descriptions derive from 1,002 sentences, with ten drawn scenes per sentence; coarse-grained paraphrasing distinguishes semantic classes, while fine-grained paraphrasing distinguishes sentences within one class.Scenes drawn from the same sentence form a semantic class and have descriptions with similar meanings.Source: Non-Visual Tasks · p. 11
  • In coarse-grained paraphrasing, Text + Visual reaches 86.15% versus 84.19% for Text only (+1.96% visual improvement), above 33.33% Random.Visual features also improve on the text-only baseline in the fine-grained setting.Source: Non-Visual Tasks · p. 12

D. Jenny is near balloons. Answers

The excerpt presents qualitative fill-in-the-blank examples and reports that, in Scenario 2, humans and the proposed approach are correct while the text baseline is incorrect.

D. Jenny is near balloons. Answers

  • Additional fragments show “B. The cat is sitting next to Jenny” and “C. Jenny is angry because it is raining on her.”A separate prompt fragment reads “Question A. Mike sees a pie.”Source: D. Jenny is near balloons. Answers · p. 13
  • Other displayed answer statements include “C. Mike and Jenny are sitting next a fire” and “D. Jenny is playing in the sandbox.”The excerpt also shows a sandbox-related fragment about Jenny, the cat, and Mike.Source: D. Jenny is near balloons. Answers · p. 13
  • In Scenario 2, humans and the proposed approach are correct while the text baseline is incorrect.The passage labels this a qualitative fill-in-the-blank result.Source: D. Jenny is near balloons. Answers · p. 14
  • For a duck-related blank, the displayed choices include “B. The ball hits the duck” and “C. The sun is shining.”The prompt mentions Mike and Jenny being scared of the duck, followed by “Happy duck walks away.”Source: D. Jenny is near balloons. Answers · p. 14

D. Mike is helping Jenny. Answers

The qualitative fill-in-the-blank examples show cases where the proposed approach is incorrect despite correct human answers, including one scenario where the text baseline is also correct.

Scenario 3

  • In Scenario 3, the human and text baseline are correct while the proposed approach is incorrect.Example prompts include whether there is an apple tree behind Mike and whether Mike is flying a kite.Source: D. Mike is helping Jenny. Answers · pp. 14–15

Scenario 4

D. Jenny has a pink shovel. Answers

The qualitative fill-in-the-blank examples show opposite outcomes: the human alone is correct in one scenario, while both model systems are correct and the human is incorrect in another.

D. Mike is very surprised. Answers

This section presents qualitative visual-paraphrasing examples comparing the proposed approach with human judgments and the text baseline. The examples show mixed outcomes, with the approach sometimes succeeding and sometimes failing alongside or differently from the other judgments.

D. Mike is very surprised. Answers

Scenario 3

  • In Scenario 3, the human and text baseline are correct while our approach is incorrect.Examples include a barbecue scene with a dog and a scene where Mike bumps into Jenny and she is surprised by Mr. Dog.Source: D. Mike is very surprised. Answers · pp. 24–25

Scenario 4

Scenario 5

  • In Scenario 5, our approach and the text baseline are correct while the human is incorrect.Examples include scenes involving Jenny and swings, a duck and a soccer ball, and Jenny with cats and a dog.Source: D. Mike is very surprised. Answers · pp. 28–29

Scenario 6

  • In Scenario 6, our approach is correct while the human and text baseline are incorrect.Examples include Jenny being upset by cats and a scene with Mike and Jenny sitting on the ground beside balls and a slide.Source: D. Mike is very surprised. Answers · pp. 30–31

Qualitative Results: Visual Paraphrasing

Related papers

Loading 1502.06108v3…