PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-supervised Visual Grounding of Phrases with Linguistic Structures

Fanyi Xiao, Leonid Sigal, Yong Jae Lee

arXiv:1705.01371v1cs.CV

TL;DR

Fine-grained visual localization often relies on costly region annotations, while image-level captions provide no explicit phrase-to-region correspondences. This paper learns pixel-level phrase masks from image-caption pairs using discriminative and parse-tree structural losses. Experiments on Microsoft COCO and Visual Genome show more accurate localizations than baselines that omit structure or use only one of its constraints.

  • Problem

    Fine-grained visual recognition often requires expensive box or pixel annotations, while image-level captions lack explicit region-to-phrase correspondences.

    Source: §1 · p. 1

  • Method

    An end-to-end model combines discriminative learning with parse-tree structural constraints on pixel-level phrase attention masks.

    Source: §1 · p. 2

  • Results

    Experiments on Microsoft COCO and Visual Genome show more accurate localizations than baselines without structure or with only one structural constraint.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Parent-child inclusivity and sibling-sibling exclusivity transfer sentence structure to visual localization in this weakly supervised setting.

    Source: §5 · p. 8

    Parser errors occur in roughly 20% of images, and limited training data constrains weak supervision relative to full supervision.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i.e., localize) arbitrary linguistic phrases, in the form of spatial attention masks. Specifically, the model is trained with images and their associated image-level captions, without any explicit region-to-phrase correspondence annotations. To this end, we introduce an end-to-end model which learns visual groundings of phrases with two types of carefully designed loss functions. In addition to the standard discriminative loss, which enforces that attended image regions and phrases are consistently encoded, we propose a novel structural loss which makes use of the parse tree structures induced by the sentences. In particular, we ensure complementarity among the attention masks that correspond to sibling noun phrases, and compositionality of attention masks among the children and parent phrases, as defined by the sentence parse tree. We validate the effectiveness of our approach on the Microsoft COCO and Visual Genome datasets.

1. Introduction

The paper addresses costly fine-grained visual annotations by learning phrase localization from image-level descriptions. It uses parse-tree structure to constrain pixel-level attention masks alongside discriminative learning.

1. Introduction

  • Fine-grained recognition tasks often depend on expensive bounding-box or pixel-level annotations, making large annotated datasets a bottleneck.Source: §1 · p. 1
  • Unlike category-tag supervision, natural-language descriptions retain semantic and structural relationships that can constrain phrase localization.For example, a sentence can indicate that a man and a sandwich occupy distinct regions, while their joint grounding spans the sentence.Source: §1 · p. 1
  • The approach learns pixel-level attention masks from image-level descriptions without region-to-phrase annotations, using parse trees to impose parent-child compositionality and sibling complementarity.Source: §1 · pp. 1–2
  • An end-to-end network combines discriminative and structural losses to encode image-phrase pairs consistently and enforce linguistic constraints on localized masks.At test time, it can generate an attention mask for an arbitrary phrase without access to the input’s linguistic structure.Source: §1 · p. 2
  • Related weakly supervised phrase-localization methods treat language as a word sequence and use bounding boxes, while this work exploits hierarchical structure and produces pixel-level masks.Source: §1 · p. 2
  • Localization experiments on Microsoft COCO and Visual Genome demonstrate the approach’s effectiveness.Source: §1 · p. 2

2. Related work

Prior weakly supervised visual learning commonly uses image-level category tags, while vision-language attention often supports captioning or question answering rather than localization as its goal. Existing phrase-localization work includes both strongly supervised methods and weakly supervised approaches that do not explicitly use sentence structure.

Weakly-supervised learning with categorical labels/tags

  • Weakly supervised detection and segmentation commonly use image-level categorical tags, whereas this work uses sentence-level supervision to retain linguistic structure.Source: §2 · p. 2

Vision and language

  • Captioning and visual question answering use attention to focus on relevant regions, but localization is typically not their final goal or explicit optimization target.Source: §2 · pp. 2–3
  • Most phrase-localization work relies on strong phrase-region supervision; Rohrbach et al. [37] use weakly annotated pairs but treat phrases as token sequences.Source: §2 · p. 3

Structure from language

  • Prior work uses linguistic structure for other tasks or under strong supervision, while this paper targets weakly supervised visual localization with hierarchical sentence structure.Source: §2 · p. 3

3. Approach

The approach learns pixel-level phrase localizations from image-sentence pairs without region-to-phrase annotations, using discriminative and parse-tree-based structural losses. It transfers linguistic relationships among phrases into constraints on their attention masks.

3. Approach

  • The method uses parse-tree structure to guide weakly supervised phrase localization, alongside image-phrase association learning.It targets pixel-level spatial attention masks without explicit region-to-phrase correspondence annotations.Source: §3 · p. 3, §3.3.1 · p. 4

3.1. Transforming a sentence into a parse tree

  • Sentences are converted to parse trees, and only nodes containing at least one noun are considered for grounding.The paper gives an off-the-shelf NLP parser as the means of producing the tree.Source: §3.1 · p. 3

3.2. Network architecture

  • Visual and language encoders project image and phrase representations into a shared semantic space, while spatial features produce phrase attention masks.Image-phrase scores feed the discriminative loss; attention masks feed the structural constraints.Source: §3.2 · pp. 3–4

3.3. Loss for weakly-supervised visual grounding

  • End-to-end training combines a structural loss on attention masks with a discriminative loss on positive and negative image-phrase pairs.The discriminative loss brings positive pairs closer and separates negative pairs in the semantic embedding space.Source: §3.3 · p. 4, §3.3.2 · p. 5

3.3.1 Structural loss

  • Parent-child constraints align a parent mask with the union of its children’s masks, while sibling constraints encourage sibling masks to be exclusive.The parent-child constraint is applied to immediate children, with transitivity carrying it through descendants.Source: §3.3.1 · pp. 4–5
  • The sibling loss weights exclusivity by average attention, emphasizing pixels where sibling masks have high values and avoiding inflation at irrelevant pixels.The structural loss combines parent-child and sibling terms with balancing weights.Source: §3.3.1 · pp. 4–5

3.3.2 Discriminative loss

  • The discriminative loss uses dot-product affinity in the semantic space to bring matching image-phrase codes together and separate mismatched pairs.Positive phrases come from the input image’s parse tree; negatives are sampled from another image’s parse tree.Source: §3.3.2 · p. 5

4. Results

The full model outperforms baselines on Visual Genome pointing-game accuracy and MS COCO segmentation. Its structural constraints also yield cleaner qualitative masks, while parser errors and weak supervision remain sources of difficulty.

4. Results

  • The evaluation covers phrase grounding on Visual Genome and category-label semantic segmentation on COCO, with Token, Disc-only, PC, SIB, and the full model as baselines or variants.Token uses noun-tagged leaf nodes; Disc-only omits structural loss, while PC and SIB each use one structural constraint.Source: §4 · p. 5

4.1. Training on MS COCO

  • The experiments train model variants on image-caption pairs from COCO train2014 and evaluate localization with the pointing game and semantic segmentation.The dataset uses 82,783 train2014 and 40,504 val2014 images.Source: §4.1 · pp. 5–6

4.2. Pointing game on Visual Genome

  • On Visual Genome, the pointing game counts a hit when the attention-mask maximum falls inside the phrase's ground-truth box; PC improves over Disc-only, and combining PC with SIB gives a large boost.SIB alone performs almost the same as Disc-only, suggesting the two structural constraints are complementary.Source: §4.2 · p. 6

4.2. Pointing game on Visual Genome; 4.3. Semantic segmentation on MS COCO

  • The full model outperforms baselines on Visual Genome pointing-game accuracy and produces the most accurate MS COCO segmentations.The MS COCO comparison spans all 80 categories; the reported mAP gains hold across IOU thresholds for PC and SIB over Disc-only, with Ours improving further.Source: Table 1 · p. 6, Table 2 · p. 7, §4.3 · p. 7

4.2. Pointing game on Visual Genome

  • Visual Genome examples show more accurate, cleaner masks than Disc-only, including for phrases about trains, boats, helmets, and motorcycles.The authors report leaky baseline masks even in cases where the baseline's maximum point falls inside the box.Source: §4.2 · p. 6

4.3. Semantic segmentation on MS COCO

  • For COCO semantic segmentation, the full model still beats the noun-token baseline, although that baseline performs well when the input is only a category noun.The authors also describe their model as more general because it can localize adjective-noun and more complex referring phrases.Source: §4.3 · p. 7
  • On COCO, examples show cleaner and more accurate masks than Disc-only, while small or infrequent objects appear among typical failure cases.For the same image, different category labels produce different groundings focused on different objects.Source: §4.3 · p. 7

4.4. Discussion of failure cases

  • The authors identify parser mistakes, sentence representations that miss relevant entities, and limited weakly supervised data as sources of difficulty.They observe parse-tree errors in roughly 20% of images and hypothesize that more weakly annotated data could improve performance.Source: §4.4 · p. 8

5. Conclusion

The paper presents a weakly supervised method that learns phrase localizations from image-level captions without region-to-phrase annotations. Experiments on Microsoft COCO and Visual Genome show more accurate localizations than baselines that omit structure or use only one structural constraint.

5. Conclusion

  • The approach learns to localize arbitrary linguistic phrases from image-level captions without explicit region-to-phrase correspondence annotations.It takes image-level captions as supervision.Source: §5 · p. 8
  • Its end-to-end network uses parent-child inclusivity and sibling-sibling exclusivity as structural loss constraints.Together, the constraints transfer natural-language structure to the visual domain.Source: §5 · p. 8
  • Experiments on Microsoft COCO and Visual Genome show more accurate localizations than baselines that omit structure or use only one constraint in isolation.Source: §5 · p. 8

Related papers

Loading 1705.01371v1…