PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Encoding Scene Graphs for Image Captioning

Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai

arXiv:1812.02378v3cs.CV

TL;DR

Image captioners can reduce unseen scenes to trivial descriptions of salient objects. SGAE uses scene graphs and a shared language-prior dictionary to guide caption generation, achieving a new single-model state of the art on the Karpathy split.

  • Problem

    For unseen scenes, encoder-decoder captioners often produce trivial descriptions of salient objects rather than richer captions.

    Source: §1 · p. 1

  • Method

    SGAE learns a language-prior dictionary by reconstructing sentences through scene graphs, then shares that dictionary to guide the image-captioning encoder-decoder.

    Source: §1 · p. 2

  • Results

    127.8 CIDEr-D on the Karpathy split established a new single-model state of the art, while 125.5 on the official test server was competitive with many ensemble models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that SGAE can embed language inductive bias in a shared dictionary and improve encoder-decoder image-captioning performance.

    Source: §7 · p. 8

Abstract

from arXiv · show

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation `person on bike', it is natural to replace `on' with `ride' and infer `person riding bike on a road' even the `road' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph ($\mathcal{G}$) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image ($\mathcal{I}$) and sentence ($\mathcal{S}$). In the textual domain, we use SGAE to learn a dictionary ($\mathcal{D}$) that helps to reconstruct sentences in the $\mathcal{S}\rightarrow \mathcal{G} \rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline, where $\mathcal{D}$ encodes the desired language prior; in the vision-language domain, we use the shared $\mathcal{D}$ to guide the encoder-decoder in the $\mathcal{I}\rightarrow \mathcal{G}\rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, e.g., our SGAE-based single-model achieves a new state-of-the-art $127.8$ CIDEr-D on the Karpathy split, and a competitive $125.5$ CIDEr-D (c40) on the official server even compared to other ensemble models.

1. Introduction

SGAE brings language inductive bias into encoder-decoder image captioning by using scene graphs and a shared trainable dictionary to bridge sentence and image representations. It targets captions that go beyond listing salient objects by supporting language-like collocations and contextual inference.

1. Introduction

  • Encoder-decoder captioners can produce trivial descriptions of salient objects, while direct rule-based exploitation of language bias is ineffective because visual perception and language composition have a large gap.The motivation is that human-like collocations and contextual inference support high-level reasoning from abstracted scenes.Source: §1 · pp. 1–2
  • Scene graphs bridge symbolic reasoning and end-to-end captioning by connecting image or sentence objects, attributes, and relationships with directed edges, enabling graph structure to be embedded as vectors.The authors expect these representations to transfer language-domain inductive bias into vision-language captioning.Source: §1 · p. 2
  • SGAE reconstructs sentences through S→G→D→S, then shares trainable dictionary D in the image-captioning pipeline I→G→D→S to guide language decoding.The sentence parser is fixed and off-the-shelf; the captioning pipeline uses a visual scene-graph detector and a multimodal GCN to add visual cues missing from imperfect detection. The dictionary re-keys encoded nodes into more generic representations with smaller domain gaps.Source: §1 · p. 2
  • On MS-COCO, SGAE achieved a 7.2-point absolute improvement in CIDEr-D over an upgraded Up-Down baseline and set a new state-of-the-art single-model score of 127.8 on the Karpathy split.Source: §1 · p. 2

2. Related Work

Prior captioning work uses encoder-decoder pipelines, attention, and reinforcement-learning objectives, while related work explores semantic knowledge, scene graphs, and memory networks. SGAE uses scene graphs to integrate object, attribute, and relationship knowledge, and a memory architecture to preserve inductive bias.

Image Captioning

  • SGAE uses a scene graph to integrate object, attribute, and relationship knowledge, extending captioning methods that incorporate these elements separately.Earlier captioning advances include encoder-decoder pipelines [42], attention, and RL-based objectives [36]; prior methods incorporate object [28], attribute [50], or relationship [49] knowledge.Source: §2 · p. 2

Scene Graphs

  • Scene graphs represent objects, attributes, and pairwise relationships; SGAE parses them from images with [52] and captions with [1].Prior work uses scene graphs for image retrieval, VQA [39], and image generation [11], and studies graph generation from images [53] [52] [48] [45] and extraction from text [1] [43].Source: §2 · p. 2

Memory Networks

  • Inspired by memory networks that preserve dynamic knowledge for inference [38] [44] [41], SGAE implements memory to preserve human inductive bias for caption generation.These methods commonly use differentiable attention to extract useful knowledge from memory.Source: §2 · p. 2

3. Encoder-Decoder Revisited

The section presents the conventional CNN-attention-RNN captioning framework and motivates modifying its mapping module to use scene graphs and a shared dictionary, aiming to counter dataset bias with language inductive bias.

3. Encoder-Decoder Revisited

  • A typical captioner extracts image features with a CNN [9] [35], uses attention [2] to re-encode them dynamically for language generation, and predicts the sentence with an RNN decoder.The goal is to generate a natural-language sentence describing the input image.Source: §3 · p. 3
  • Training can minimize cross-entropy against a ground-truth caption or maximize an RL reward [36], where the sentence-level metric can be CIDEr-D [40].The reward compares a sampled sentence with the ground-truth caption.Source: §3 · p. 3
  • The encoder-decoder framework underpins nearly all state-of-the-art image captioners since [42], but is brittle to dataset bias [28], motivating language inductive bias for more human-like captions.The section characterizes dataset bias as pernicious and language inductive bias as beneficial.Source: §3 · p. 3
  • SGAE modifies the mapping module with scene graphs and a shared-dictionary re-encoder, while MGCN uses the dictionary to re-encode image features and narrow the vision-language gap.The dictionary is first learned through sentence self-reconstruction with scene graphs, then used in the overall image captioner.Source: §3 · p. 3

4. Auto-Encoding Scene Graphs

SGAE learns a dictionary through sentence self-reconstruction, transforming sentences into scene graphs and context-aware embeddings before re-encoding and decoding them. Its graph structure and dictionary provide the basis for contextual language composition.

4. Auto-Encoding Scene Graphs

  • SGAE self-reconstructs sentences with an encoder-decoder pipeline that decodes the re-encoded features X.The section describes learning the dictionary D through sentence reconstruction and gives the decoder as S ← X̂.Source: §4 · p. 3

4.1. Scene Graphs

  • Scene graphs represent objects, attributes, and relationships with directed edges and trainable 1,000-dimensional label embeddings.The sentence-to-graph parser uses [1], which builds a dependency tree with and transforms it into a scene graph using rules.Source: §4.1 · pp. 3–4

4.2. Graph Convolution Network

  • Four spatial graph convolutions produce context-aware relationship, attribute, and object embeddings, incorporating triplet context and objects’ subject/object roles.Object embeddings include whole-triplet context to distinguish predicate meanings, such as “with” in “hand-with-cup” versus “head-with-hat.”Source: §4.2 · p. 4

5. Overall Model: SGAE-based Encoder-Decoder

The SGAE-based encoder-decoder uses MGCN to adapt image features for re-encoding with the shared dictionary before generating captions.

5.1. Multi-modal Graph Convolution Network

  • MGCN maps CNN visual features to scene-graph-modulated features, which the shared dictionary re-encodes before an attention decoder generates language.The image scene graph is produced by object, attribute, and relationship classifiers; its object nodes fuse detected label embeddings with Faster-RCNN RoI features.Source: §5.1 · p. 5
  • Image-graph object, attribute, and relationship embeddings are computed by four two-layer functions, while object-node fusion combines label and visual features.The image graph is described as simpler and noisier than the sentence graph; the fusion is reported to converge faster than popular bi-linear fusion.Source: §5.1 · p. 5

5.2. Training and Inference

  • The overall model initializes its dictionary from pretrained SGAE, fine-tunes it at 10^-5, trains with cross-entropy for 20 epochs then RL reward for 40, and uses beam search of size 5.The fine-tuning learning rate is intentionally small to encourage dictionary sharing.Source: §5.2 · p. 5

6. Experiments

Experiments evaluate SGAE on MS-COCO and find its graph components and shared dictionary beneficial; the full single model reaches 127.8 CIDEr-D on the Karpathy split.

6.1. Datasets, Settings, and Metrics

  • Using VG annotations, experiments train the scene-graph parser, while captioner training uses MS-COCO text except in ablations.The captions are lowercased, filtered to words appearing at least five times, and trimmed to 16 words; VG labels are filtered to 305 objects, 103 attributes, and 64 relationships appearing more than 2,000 times.Source: §6.1 · p. 5

6.3. Comparisons with State-of-The-Arts

  • 127.8 CIDEr-D makes SGAE the highest-scoring single model among the compared methods on the Karpathy split; SGAEfuse also outperforms GCN-LSTMfuse.On the official server, SGAE’s single model achieves the highest CIDEr-D score among the published methods compared as of 16/11/2018.Source: Table 1 · p. 6, Table 4 · p. 8, §6.3 · p. 8

6.2.1 Architecture

  • Adding MGCN, GCN, or the shared dictionary improves performance over Base, and SGAE raises CIDEr-D by 7.2 points over Base.The authors report that Base+D and Base+D w/o GCN outperform Base+MGCN, supporting the role of language inductive bias; SGAE captions are more complex and descriptive than Base captions.Source: Table 1 · p. 6, §6.2.1 · p. 6, Figure 5 · p. 7

6.2.3 Sentence Reconstruction

  • Direct reconstruction from scene-graph features performs better than reconstruction re-encoded by D, whose regularization encourages language inductive bias.The gap between reconstructed and generated captions suggests that better image scene-graph parsing and a stronger re-encoder could improve results when visual graphs are low quality.Source: Table 3 · p. 6, §6.2.3 · p. 8

6.2.2 Language Corpus

  • Both VG web text and matched MS-COCO text improve over Base, while SGAE with matched text scores better than using web text.Web-text captions tend toward general expressions, matched-corpus captions toward dataset-specific expressions, and human evaluators judge captions using a dictionary more descriptive.Source: §6.2.2 · pp. 6–8

7. Conclusions

The paper proposes SGAE to embed a language prior for more human-like image captioning in a shared dictionary that supports language generation. It reports validation on MS-COCO and proposes applying SGAE to other vision-language tasks.

7. Conclusions

  • SGAE embeds language inductive bias in a dictionary shared as a re-encoder for language generation, improving the encoder-decoder image-captioning framework.The paper presents SGAE as a novel unsupervised learning method.Source: §7 · p. 8
  • The authors validate the SGAE-based framework through extensive ablations and comparisons with state-of-the-art performance on MS-COCO.Source: §7 · p. 8
  • The authors propose applying SGAE to other vision-language tasks as a general solution for capturing language inductive bias.Source: §7 · p. 8

8. Network Architecture

Section 8 details the architectures of the GCN, MGCN, dictionary re-encoder, and two decoders.

8.1. Graph Convolutional Network

  • The GCN maps one-hot object, relation, and attribute labels through a shared word-embedding matrix, then computes graph embeddings with fully connected layers and ReLUs.The word-embedding matrix has dimensions 1,000×10,102; the embedding functions share a structure but use independent parameters.Source: §8.1 · p. 11

8.2. Multi-modal Graph Convolutional Network

  • The MGCN combines object and relation RoI features from Faster R-CNN [35] with label embeddings using feature fusion to compute graph embeddings.It uses a distinct 1,000×472 word-embedding matrix, and replaces word embeddings with fused embeddings and the GCN functions with fully connected layers followed by ReLUs.Source: §8.2 · p. 11, Table 6 · p. 12
  • The MGCN specifies a relationship-embedding computation using fused embeddings at Table 6(12).The relationship RoI feature is pooled from a region covering the subject and object.Source: §8.2 · p. 11

Attribute Embedding v

  • The MGCN specifies an attribute-embedding computation using fused embeddings at Table 6(13).The attribute embedding follows the MGCN’s fully connected layer and ReLU transformation.Source: §8.2 · p. 11

Object Embedding v

8.3. Dictionary

  • The dictionary re-encoder softmax-normalizes inner products between an index vector and dictionary atoms, then returns their weighted sum.This operation re-encodes a representation from the index vector and dictionary.Source: §8.3 · p. 12, Table 7 · p. 12

8.4. Decoders

  • Both decoders follow [2] and share two LSTM layers and an attention module, which selects embeddings from Z to produce each next-word distribution.The first LSTM receives the previous word embedding, mean-pooled embeddings, and prior second-LSTM output; the SGAE decoder uses re-encoded X, while the SGAE-based encoder-decoder uses scene-graph-modulated and re-encoded visual features.Source: §8.4 · pp. 12–13

9. Details of Scene Graph

The section describes how sentence and image scene graphs are parsed, filtered, and represented as continuous vectors. Image graphs use object, relationship, and attribute predictions with specified thresholds and embedding dimensions.

9.1. Sentence Scene Graph

  • Sentence graphs are parsed with software [1], filtered to remove labels occurring fewer than 10 times, and represented using the word embedding matrix WΣS in Table 5.After filtering, 5,364 objects, 1,308 relationships, and 3,430 attributes remain.Source: §9.1 · p. 13

9.2. Image Scene Graph

  • Image graphs use Faster-RCNN [35] for object detection, an LSTM relationship classifier [52], and an attribute classifier that predicts the top three attributes per object.The RoI features also serve as input to the relationship classifier, attribute classifier, and MGCN.Source: §9.2 · p. 13
  • Image detection keeps 10–100 objects per image, applies proposal and object NMS IoU thresholds of 0.7 and 0.3, and predicts relationships for object pairs with IoU above 0.2.Faster-RCNN training followed [2].Source: §9.2 · p. 13
  • Image scene graphs use 472 objects, relationships, and attributes represented by a 472 × 1,000 word embedding matrix, as in Table 6 (6) to (8).Source: §9.2 · p. 14

10. More Qualitative Examples

The qualitative examples suggest that SGAE favors more accurate descriptions of objects, attributes, relationships, and scenes, including details not accurately recognized or evident in the image.

10. More Qualitative Examples

  • SGAE uses more accurate object and attribute descriptions, including “weather vane” despite detector error and “old rusty” in Figure 8.These are examples from the qualitative comparison with baselines.Source: §10 · p. 14
  • The authors report that in Figure 9, SGAE describes the boat’s relationship with water as floating instead of swimming, and infers mountains as the scene.Figure 9 shows 12 qualitative examples of different baselines.Source: §10 · p. 14, Figure 9 · p. 15

Related papers

Loading 1812.02378v3…