PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, Xilin Chen

arXiv:2003.13962v1cs.CV

TL;DR

Rare, ambiguous scene words challenge image-question answering because pretrained word embeddings alone are insufficient. MM-GNN represents visual, semantic, and numeric content in separate graphs and passes messages between them; the paper reports better scene-text representations and facilitated performance on two text-reading VQA tasks.

  • Problem

    Scene texts combine visual, semantic, and numeric information, while rare or polysemous words can be difficult to interpret from pretrained word embeddings alone.

    Source: §1 · pp. 1–2

  • Method

    MM-GNN represents visual, semantic, and numeric image content as three subgraphs and uses attention-based aggregators to pass contextual messages and refine node features.

    Source: §1 · p. 2

  • Results

    MM-GNN effectively learns scene-text representations and facilitates performance on TextVQA and ST-VQA.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The approach improves VQA with scene texts and provides interpretable intermediate results.

    Source: §5 · p. 8

    The Numeric-Semantic aggregator contributes less than the other two, possibly because relatively few questions query number relations.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Answering questions that require reading texts in an image is challenging for current models. One key difficulty of this task is that rare, polysemous, and ambiguous words frequently appear in images, e.g., names of places, products, and sports teams. To overcome this difficulty, only resorting to pre-trained word embedding models is far from enough. A desired model should utilize the rich information in multiple modalities of the image to help understand the meaning of scene texts, e.g., the prominent text on a bottle is most likely to be the brand. Following this idea, we propose a novel VQA approach, Multi-Modal Graph Neural Network (MM-GNN). It first represents an image as a graph consisting of three sub-graphs, depicting visual, semantic, and numeric modalities respectively. Then, we introduce three aggregators which guide the message passing from one graph to another to utilize the contexts in various modalities, so as to refine the features of nodes. The updated nodes have better features for the downstream question answering module. Experimental evaluations show that our MM-GNN represents the scene texts better and obviously facilitates the performances on two VQA tasks that require reading scene texts.

1. Introduction

Scene texts combine visual, semantic, and numeric information, while rare or ambiguous words make semantic interpretation especially difficult. MM-GNN models these modalities with separate graphs and attention-based aggregators; experiments on TextVQA and ST-VQA report better scene-text representations and facilitated question answering.

1. Introduction

  • The intended application includes daily tasks involving scene text, such as finding places or learning about products, and assistance for visually impaired users and children’s education.The paper focuses on improving VQA for questions that require reading text in images.Source: §1 · p. 1
  • Scene texts combine visual, semantic, and numeric information, and rare or polysemous words make their semantics difficult to determine from pretrained word embeddings alone.The introduction motivates using context across modalities, such as a prominent bottle label suggesting a brand.Source: §1 · pp. 1–2, Figure 1 · p. 1, Figure 2 · p. 2
  • MM-GNN represents an image with visual, semantic, and numeric subgraphs, initializing node representations from priors such as word embeddings and Faster R-CNN features.The visual graph includes texts and objects; the semantic graph represents scene texts, and the numeric graph represents number-related texts.Source: §1 · p. 2
  • Three attention-based aggregators use visual appearance, layout, and question cues to pass messages across or within modalities and refine node features.The resulting representations are intended to help the answering model attend to the correct answer.Source: §1 · p. 2
  • MM-GNN effectively learns scene-text representations and facilitates question answering on TextVQA and ST-VQA.The reported evaluation covers two datasets requiring reading text in images.Source: §1 · p. 2

2. Related Work

Related work spans visual-information VQA, compositional spatial reasoning, and progressively richer image representations. This paper focuses on text-reading VQA and extends graph-based representation and reasoning to multimodal image contents.

2. Related Work

  • VQA datasets include questions about visual information and, in CLEVR and GQA, complex compositional spatial reasoning.Related approaches include attention and multimodal fusion, modular networks, and neural-symbolic models [2] [3] [20] [25] [22] [43] [51].Source: §2 · p. 2
  • TextVQA and ST-VQA ask about images containing visual objects and diverse scene texts, motivating better representations of both scene texts and objects.The paper frames these as a distinct type of VQA question compared with other tasks.Source: §2 · p. 2
  • Image representations progressed from grid-level features, limited for object-level attention, to detected-object features, symbolic scene graphs, and fully connected graphs.The cited fully connected graph approaches dynamically weight edges to focus on question-related subgraphs [36] [32] [21].Source: §2 · pp. 2–3
  • MM-GNN extends visual-object and single-modal graph representations to three connected modality-specific subgraphs with aggregators for distinct multimodal contexts.It builds on grounding-task GNNs [45] [36] [32] [21] while explicitly capturing multimodal context in its aggregation schemes.Source: §2 · p. 3

3. Method

MM-GNN represents an image with visual, semantic, and numeric subgraphs, then uses successive message passing to refine node representations for question answering. Its answer module predicts from the updated graph features.

3.1. Multi-Modal Graph Construction

  • The multimodal graph contains visual, semantic, and numeric subgraphs; visual and numeric graphs are fully connected, visual nodes encode entities, semantic tokens start with pretrained word embeddings, and numeric texts receive typed encodings.Source: §3.1 · p. 3

3.2. Aggregation Scheme

  • Visual-Semantic aggregation uses question-, appearance-, and layout-informed attention to refine text nodes from visual neighbors and visual carriers from semantic context.The visual carriers are the objects on which texts are printed or carved.Source: §3.2 · p. 4
  • Semantic-semantic attention adds neighboring token context, while semantic-numeric attention transfers semantic-node information to numeric nodes.Semantic-node relevance incorporates token representations and image locations.Source: §3.2 · pp. 4–5

3.3. Answer Prediction

  • The answer module scores candidates using updated visual and OCR features, with an output space that allows answers to be copied from OCR tokens.Binary cross-entropy training does not penalize either route when an answer appears in both the predefined answer space and OCR tokens.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate MM-GNN on TextVQA and ST-VQA, reporting strong comparisons with existing methods and examining graph components, combination schemes, and attention behavior. Ablations and qualitative examples probe how the model uses multimodal context to answer scene-text questions.

4.1. Experiments Setup

  • On TextVQA and ST-VQA, the setup uses dataset-provided OCR for TextVQA and TextSpotter-extracted scene text for ST-VQA, with VQA accuracy and ANLS/accuracy evaluation, respectively.TextVQA uses Rosetta OCR, while the ST-VQA experiments use Faster R-CNN image features and the same question and OCR-token embedding models as TextVQA.Source: §4.1 · p. 5

4.2. Results

  • MM-GNN achieves state-of-the-art results on TextVQA and improves over ST-VQA baselines while reaching accuracies comparable to VTA.Table 1 reports TextVQA VQA accuracy; Table 2 covers ANLS and accuracy on ST-VQA's Weakly Contextualized and Open Dictionary tasks.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • On TextVQA validation, MM-GNN beats No-GNN by about 4% overall and over 8% on OCR-related questions; Visual-Semantic contributes most, while Vanilla GNN offers little help.The authors interpret this as evidence that graph representation helps answering when message passing is designed for multimodal contexts.Source: §4.2 · pp. 6–7, Table 3 · p. 7
  • Numeric-Semantic contributes less than the other aggregators, which the authors suggest may reflect the small share of questions about numeric relations.They say this limits the opportunity to demonstrate that aggregator's effectiveness.Source: §4.2 · p. 7
  • Sum, Product, and Concat + MLP combination variants all reduce TextVQA validation performance, which the authors attribute to compressing neighborhood information and blurring node-feature differences.Their concatenation updater avoids the fixed node-feature-size restriction of those alternatives.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. Qualitative Analysis

  • Attention visualizations show OCR answers drawing on relevant objects and neighboring tokens, including carrier links and contextual cues for interpreting unfamiliar text.Examples include using nearby “BREAD” to interpret “Panera” and surrounding sign text and OCR context to infer that “transilvania” is a bank name.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

5. Conclusion

MM-GNN represents images with scene text as three modality-specific graphs and uses multi-modal aggregators to refine image-element representations. Experiments report improved scene-text VQA performance and interpretable intermediate results.

5. Conclusion

  • MM-GNN represents image content as three graphs, each corresponding to one modality.The framework targets VQA with scene texts.Source: §5 · p. 8
  • Multi-modal aggregators use cross-modal contexts to refine representations, especially for unknown, rare, or polysemous words.Source: §5 · p. 8
  • Experiments report improved scene-text VQA performance and interpretable intermediate results.Source: §5 · p. 8

Overview

The supplementary material provides further implementation details and additional quantitative and qualitative experimental analyses.

Overview

  • The supplement provides further implementation details for the number encoder used in graph construction.Source: Overview · p. 12
  • Additional quantitative experimental results are presented in the supplementary material.Source: Overview · p. 12
  • The supplement also includes qualitative analysis of MM-GNN.Source: Overview · p. 12

6. Number encoder

The number encoder treats digital numbers and days of the week as numeric strings, including strings with periodic information. It normalizes periodic numbers before applying cosine embedding, while noting that numeric-string encoding remains an open problem.

6. Number encoder

  • Digital numbers and days of the week are treated as numeric-type strings, like time-related strings with periodic information.Source: §6 · p. 12
  • A periodic number such as 10:00 is normalized to 10/24 before applying the cosine embedding x → cos 2πx.Source: §6 · p. 12
  • Numeric-string encoding remains open; the paper notes that a polar-coordinate representation using cosine and sine could better represent periodic numbers.Source: §6 · p. 12

7. Quantitative Analysis

The analysis reports stronger MM-GNN performance than No-GNN mainly on TextVQA questions about diverse scene texts, and similar performance across aggregator orders.

7. Quantitative Analysis

  • The evaluated order is Visual-Semantic, then Semantic-Semantic, followed by Semantic-Numeric aggregation.Source: §7 · p. 12
  • Different aggregator orders yield similar performance, indicating that MM-GNN is robust to order changes.Source: Table 5 · p. 12, §7 · p. 12
  • MM-GNN mainly outperforms No-GNN on TextVQA others-type questions, which mostly concern understanding diverse scene texts.Source: Table 6 · p. 12, §7 · p. 12

8. Qualitative Analysis

The qualitative analysis presents successful TextVQA cases with correct answers and reasonable attention, alongside failure cases with possible reasons analyzed.

8. Qualitative Analysis

  • MM-GNN answers the successful TextVQA examples correctly, with reasonable attention when using multimodal contexts.Figure 6 shows the successful cases.Source: §8 · p. 12, Figure 6 · p. 13
  • The analysis also presents MM-GNN failure cases and possible reasons for each example.Figure 7 shows the failure cases.Source: §8 · p. 12, Figure 7 · p. 14
  • Figure 6 maps OCR-token attention to five visual objects and between OCR tokens to two other tokens; bolder boxes indicate higher attention.Visual-object boxes are red and OCR-token boxes are green.Source: Figure 6 · p. 13

Related papers

Loading 2003.13962v1…