PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Encoding Scene Graphs for Image Captioning

Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai

arXiv:1812.02378v3cs.CV

TL;DR

Image captioning systems can produce trivial, dataset-biased descriptions, motivating a way to add language inductive bias without abandoning end-to-end multimodal mapping. The paper introduces SGAE, which learns a shared dictionary from sentence scene-graph reconstruction and uses it to guide image-to-caption generation. On MS-COCO, the method improves over a strong baseline and achieves reported state-of-the-art or competitive CIDEr-D scores.

  • Problem

    Image captioning models often generate trivial salient-object descriptions and are brittle to dataset bias, despite strong visual recognition capabilities.

  • Method

    SGAE learns language inductive bias through sentence self-reconstruction over scene graphs and a trainable dictionary, which is shared to guide image captioning with multimodal graph convolution.

  • Results

    127.8 CIDEr-D on the Karpathy split and 125.5 on the official test server are reported for SGAE-based captioning, with a 7.2-point CIDEr-D improvement over an upgraded Up-Down baseline.

  • Takeaways & Limitations

    Scene graphs and a shared dictionary transfer language inductive bias from sentence reconstruction into vision-language caption generation.

Abstract

from arXiv · show

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation `person on bike', it is natural to replace `on' with `ride' and infer `person riding bike on a road' even the `road' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph ($\mathcal{G}$) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image ($\mathcal{I}$) and sentence ($\mathcal{S}$). In the textual domain, we use SGAE to learn a dictionary ($\mathcal{D}$) that helps to reconstruct sentences in the $\mathcal{S}\rightarrow \mathcal{G} \rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline, where $\mathcal{D}$ encodes the desired language prior; in the vision-language domain, we use the shared $\mathcal{D}$ to guide the encoder-decoder in the $\mathcal{I}\rightarrow \mathcal{G}\rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, e.g., our SGAE-based single-model achieves a new state-of-the-art $127.8$ CIDEr-D on the Karpathy split, and a competitive $125.5$ CIDEr-D (c40) on the official server even compared to other ensemble models.

1. Introduction

The paper targets trivial, dataset-biased captions by transferring language inductive bias through scene graphs and a shared dictionary into encoder-decoder image captioning. SGAE reconstructs sentences to learn this bias, while multimodal graph processing and the shared dictionary guide visual caption generation.

  • Motivation: Modern encoder-decoder captioners often produce trivial descriptions of salient objects, resembling object-detection lists despite advances in visual recognition.The paper identifies dataset bias as a persistent weakness of the prevailing framework.
  • Motivation: Human-like captioning uses symbolic scene abstraction, collocations, and contextual inference as a language inductive bias.The paper motivates transferring this bias from language into vision-language generation.
  • Approach: Scene graphs unify objects, attributes, and relationships in images and sentences, providing a bridge between symbolic reasoning and multimodal feature mapping.Graph convolutional networks embed this structure into vector representations that can integrate with encoder-decoder models.
  • Approach: SGAE learns a trainable dictionary by reconstructing sentences through S → G → D → S, then shares D in the image-captioning pipeline I → G → D → S.A language parser and RNN decoder support sentence reconstruction, while visual graph detection and a multimodal GCN process images.
  • Results: 7.2 CIDEr-D points improve over an upgraded Up-Down baseline, while the SGAE single model reaches 127.8 on the Karpathy split and 125.5 on the official test server.The reported results are on MS-COCO image captioning.
  • Contributions: The contributions comprise SGAE language-bias representation learning, multimodal graph convolution for visual scene graphs, and shared-dictionary guidance for decoding.These components define the proposed SGAE-based encoder-decoder captioner.

2. Related Work

The related work situates SGAE among image-captioning methods that progressively add semantic structure and among scene-graph approaches for vision tasks. Its distinction is using scene graphs to integrate objects, attributes, and relationships as semantic context.

  • Image Captioning: Modern image captioning builds on encoder-decoder pipelines, attention, and reinforcement-learning objectives, while earlier systems relied on rules or templates.Later methods also incorporate object, attribute, or relationship knowledge.
  • Image Captioning: SGAE uses scene graphs as a bridge that integrates object, attribute, and relationship knowledge for more meaningful caption contexts.The paper contrasts this unified representation with approaches using these semantic elements separately.
  • Model Comparison: Figure 2 contrasts the conventional encoder-decoder with the proposed architecture, where SGAE embeds language inductive bias in a shared dictionary.The proposed design adds the shared-dictionary language prior to the conventional framework.

3. Encoder-Decoder Revisited

The standard encoder-decoder captioning pipeline maps image features to language but remains brittle to dataset bias. The paper introduces scene-graph re-encoding with a shared dictionary to inject language inductive bias into this pipeline.

  • Image captioning encodes an image into visual representations and decodes them into a language sequence.
  • Reinforcement-learning training can optimize sequence generation using a sentence-level metric such as CIDEr-D.
  • The encoder-decoder framework is widely used but is brittle to dataset bias.
  • The proposed framework modifies the visual feature mapping module by introducing a scene graph and re-encoder parameterized by a shared dictionary.

4. Auto-Encoding Scene Graphs

The Scene Graph Auto-Encoder reconstructs sentences through scene graphs and a trainable dictionary that re-encodes contextual node features. This dictionary is intended to embed language inductive bias and support contextual inference during reconstruction.

  • Auto-Encoding Scene Graphs: SGAE reconstructs a sentence through the S → G → D → S pipeline, using a scene graph parser and an RNN-based decoder.
  • Scene Graphs: A scene graph contains object, attribute, and relationship nodes connected by directed edges.
  • Graph Convolution Network: Graph convolutions transform object, attribute, and relationship label embeddings into context-aware node representations.
  • Dictionary: The re-encoder represents each node feature as a dictionary-weighted combination whose weights are computed by softmax(D^T x).
  • Dictionary: Re-encoding can make generated language reflect contextual associations, such as changing “yellow and dotted banana” toward “ripe banana.”
  • Dictionary: SGAE training is unsupervised and reconstructs sentences end-to-end using cross-entropy or reinforcement-learning objectives.

5. Overall Model: SGAE-based Encoder-Decoder

The overall model maps image features into scene-graph-modulated features before applying the shared dictionary re-encoder and language decoder. It combines multimodal graph convolution with dictionary initialization and staged training.

  • Overall Model: The overall SGAE-based encoder-decoder uses a Multi-modal Graph Convolution Network to map visual features into scene-graph-modulated features.
  • Visual Scene Graph: Image scene graphs are extracted using object detection, attribute classification, and relationship classification.
  • Visual Scene Graph: Visual and detected-label features are fused into node features before multimodal graph convolution.
  • Overall Model: The re-encoded visual features are fed to the attention-based decoder to generate language.
  • Training: The pretrained dictionary is fine-tuned with a small learning rate, using cross-entropy for 20 epochs followed by reinforcement-learning reward for 40 epochs.
  • Inference: Inference uses beam search with beam size 5.

6. Experiments

Experiments on MS-COCO evaluate SGAE through dataset comparisons, ablations, qualitative examples, human evaluation, and state-of-the-art benchmarking. The results support the importance of the shared dictionary, graph components, and language-corpus choice for descriptive captioning.

  • Datasets and settings: MS-COCO experiments use the official online test split and the Karpathy split, with five human-labeled captions per image.
  • Ablations: SGAE improves CIDEr-D by 7.2 absolute points over the Base baseline on the MS-COCO Karpathy split.
  • Ablations: MGCN, GCN, and the shared dictionary D each improve performance, while dictionary-based variants outperform Base+MGCN.
  • Language corpus: Web and MS-COCO language corpora both improve captions over Base, with matched corpora producing dataset-specific expressions and Web texts producing more general ones.
  • Qualitative and human evaluation: Human evaluations find captions generated with D more descriptive, and qualitative examples show SGAE producing more complex descriptions of attributes and relationships.
  • Sentence reconstruction: Direct scene-graph features reconstruct sentences better than dictionary-re-encoded features, indicating that D regularizes reconstruction toward language inductive bias.
  • State-of-the-art comparison: 127.8 CIDEr-D makes the SGAE single model state of the art among compared methods, while the official-server model achieves competitive performance.

7. Conclusions

The paper introduces SGAE to embed language inductive bias in a shared dictionary and use it within encoder-decoder image captioning. Extensive MS-COCO ablations and comparisons validate the framework’s performance, with planned extension to other vision-language tasks.

  • SGAE learns a dictionary that embeds language inductive bias from sentence self-reconstruction and shares it as a re-encoder for language generation.
  • Extensive MS-COCO ablations and state-of-the-art comparisons validate the SGAE-based image-captioning framework.
  • The authors plan to apply SGAE to other vision-language tasks as a general solution for capturing language inductive bias.

8. Network Architecture

The architecture combines graph convolutional networks, multimodal visual features, a dictionary re-encoder, and LSTM-attention decoders. The decoders differ by whether they receive sentence features or combined scene-graph-modulated and re-encoded visual features.

  • Graph Convolutional Network: GCN computes three embeddings from sentence scene graphs using object, relation, and attribute labels mapped through a word-embedding matrix.
  • Multi-modal Graph Convolutional Network: MGCN combines object and relation labels with RoI features extracted by a pretrained Faster R-CNN to represent visual scene graphs.
  • Dictionary: The dictionary re-encoder computes similarities between an index vector and dictionary atoms, normalizes them with softmax, and returns their weighted sum.
  • Decoders: Each decoder has two LSTM layers and an attention module that produces a word distribution from accumulated context and attended embeddings.
  • Decoder inputs: SGAE uses re-encoded sentence features, whereas the SGAE-based encoder-decoder uses concatenated scene-graph-modulated and re-encoded visual features.

9. Details of Scene Graph

This section details the scene-graph construction and illustrates caption differences across baselines. SGAE produces richer descriptions than the base models in the shown examples.

  • Scene-graph construction: Sentence scene graphs retain objects, relationships, and attributes occurring at least 10 times, leaving 5,364 objects, 1,308 relationships, and 3,430 attributes.The remaining node labels are transformed into continuous vectors using the word-embedding matrix WΣS.
  • Scene-graph construction: Image scene graphs are built from Faster-RCNN object detection, MOTIFS relationship classification, and an attribute classifier.These components predict the objects, relationships, and attributes used to construct each image graph.
  • Qualitative comparison: In the truck examples, SGAE generates “an old rusty truck” that is “parked,” whereas BASE and BASE+MGCN provide less specific descriptions.The ground-truth caption also includes the attributes “rusty old” and the truck’s parked context.
  • Evaluation resources: The scene-graph examples and parsed graphs support qualitative evaluation of generated captions and are accompanied by a planned code release.Figures 8 and 9 use pruned image scene graphs to avoid clutter, while the codes and parsed graphs are intended for further research.
  • Scene-graph construction: The image-graph vocabulary contains 472 objects, relationships, and attributes, represented with a 472 × 1,000 word-embedding matrix.The matrix transforms predicted node labels into continuous vectors.

10. More Qualitative Examples

The qualitative examples show SGAE selecting more accurate object, attribute, and relationship words and inferring scene content that other captions omit.

  • Object descriptions: SGAE uses “weather vane” even when the object detector does not accurately recognize that object.This example illustrates a caption using a more accurate object description than the detector output alone.
  • Attribute descriptions: SGAE describes the truck attribute as “old rusty,” adding specificity beyond the baseline captions.The example contrasts SGAE’s attribute choice with less detailed alternatives.
  • Relationships and scenes: SGAE describes a boat’s relation to water as “floating” rather than “swimming” and infers the scene “mountains.”These examples cover both relationship selection and scene-level inference.
Loading 1812.02378v3…