PaperlayerView exact revision on arXiv

Source-linked AI summary

SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning

Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, Parisa Kordjmashidi

arXiv:2104.05832v1cs.CLcs.AI

TL;DR

Textual spatial reasoning has limited coverage in prior datasets, while the available bAbI task is synthetic and simplified. The paper introduces human-curated and automatically generated spatial QA data, then reports improved performance after further pretraining, including gains on bAbI and boolQ.

  • Problem

    Textual spatial reasoning has limited dataset coverage, and bAbI Task 17 is synthetic with simple scenes, short templated stories, and usually shallow reasoning.

    Source: §1 · p. 1

  • Method

    The paper creates a human-curated spatial QA benchmark and generates additional spatial descriptions and QA pairs from visual scenes using grammars and spatial reasoning rules.

    Source: §1 · pp. 1–2

  • Results

    Further pretraining improves performance on SPARTQA-HUMAN and transfers to bAbI and boolQ; BERT achieves 99% bAbI accuracy with half the original BERT training data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPARTQA-AUTO provides spatial knowledge that improves language-model performance on the human benchmark and on bAbI and boolQ.

    Source: §1 · p. 2

    The transfer approach is limited to further pretraining, with more advanced methods left for future work.

    Source: §1 · p. 2

Abstract

from arXiv · show

This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.

1 Introduction

The paper addresses limited textual spatial reasoning resources by introducing SPARTQA, a benchmark with richer scenes, stories, and questions than prior synthetic data. It also uses automatically generated spatial QA for further pretraining, reporting improved performance on the benchmark and two external datasets.

1 Introduction

  • Prior textual spatial QA was limited: bAbI Task 17 is described as synthetic, with simple scenes, templated short stories, and questions usually requiring at most two reasoning steps.The introduction also notes limited investigation of spatial reasoning in language models.Source: §1 · p. 1
  • SPARTQA uses NLVR images, more natural stories, richer spatial relations, and questions requiring deeper reasoning across four types.Its question types are find relation, find blocks, choose object, and yes/no.Source: §1 · p. 1
  • The authors generate SPARTQA-AUTO from NLVR images using context-free grammars and context-sensitive rules, recording objects and relations to create QA pairs with spatial-rule inferences.The generated data provide distant supervision for spatial reasoning over text.Source: §1 · pp. 1–2
  • Further pretraining on SPARTQA-AUTO improves performance on SPARTQA-HUMAN and transfers to bAbI and boolQ; BERT reaches 99% bAbI accuracy with half the original BERT training data.On boolQ’s development set, the model also achieves 2.3% relative error reduction compared with BERT.Source: §1 · p. 2
  • The paper uses further pretraining as a baseline knowledge-transfer method and leaves more advanced methods for future work.The authors characterize further pretraining as common practice for transferring knowledge between tasks.Source: §1 · p. 2
  • SPARTQA-HUMAN is presented as the first human-curated spatial reasoning benchmark, with richer phenomena than synthetic bAbI Task 17.The paper describes a small-scale set of 1.1k QA pairs, evenly distributed across four question types.Source: §1 · pp. 1–2

2 Related work

Prior work has covered spatial reasoning mainly in synthetic textual data or in multimodal and other grounded settings. The paper focuses instead on spatial reasoning expressed in natural-language text.

2 Related work

  • Other QA research tests linguistic phenomena, but spatial reasoning has received minimal coverage despite QA’s flexibility for annotating complex phenomena in natural language.A separate synthetic dataset evaluates generation of spatial facts and sentences involving composition and decomposition of relations on grounded objects.Source: §2 · pp. 2–3
  • bAbI Task 17 is identified as the only prior QA dataset focused on textual spatial reasoning, but its synthetic stories and simple questions do not reflect natural-language complexity.The paper notes that solving it typically requires little sophisticated reasoning.Source: §2 · p. 3
  • Spatial reasoning appears in multimodal QA benchmarks such as NLVR, VQA, GQA, and CLEVR, where it is mostly conveyed through images rather than natural-language text.Other cited work addresses visual-spatial reasoning using maps, diagrams, and navigational instructions.Source: §2 · p. 3

3 SPARTQA-HUMAN

SPARTQA-HUMAN uses NLVR images to elicit natural descriptions of spatial scenes and questions that require reasoning over described relations. Its annotations intentionally omit some information, and the resulting small dataset contains 1.1k QA pairs.

3 SPARTQA-HUMAN

  • The annotation process starts from NLVR images with more objects and less restricted spatial arrangements than the simple scenes described by bAbI.The images contain three blocks, and objects can occupy arbitrary positions within them.Source: §3 · p. 3
  • A question about whether the square is left of the blue circle receives Do not Know when the description gives no relation between them.An unstated relation is not treated as false.Source: Figure 2 · p. 3, §3 · p. 4
  • Annotators described objects and relations naturally, including relative direction, qualitative distance, and topological relationships, and could rearrange horizontally aligned blocks.They were instructed not to describe every object or relation, leaving information unstated to support more complex reasoning.Source: §3 · p. 3
  • Annotators were encouraged to ask questions requiring multiple spatial rules, including transitivity, symmetry, converse, inclusion, and exclusion.These rules support inference beyond relations stated directly in the descriptions.Source: §3 · p. 3
  • The benchmark has four question types: find relation, find blocks, choose object, and yes/no.The first three are multiple-choice; yes/no answers include Yes, No, and Do not Know under an open-world assumption.Source: §3 · p. 3
  • The dataset contains 1.1k QA pairs distributed across the question types, and experts averaged 92% accuracy on 100 test examples.The authors kept the dataset relatively small; annotation cost about 100 human hours, and the authors cite shortcutting concerns as well as collection effort.Source: §3 · p. 4

4 Distant Supervision: SPARTQA-AUTO

SPARTQA-AUTO generates stories and question-answer pairs from NLVR scenes using grammars and spatial reasoning rules. Its question-generation process uses stored scene relationships and several question types to exercise different reasoning patterns.

4 Distant Supervision: SPARTQA-AUTO

  • SPARTQA-AUTO generates stories and QA pairs from NLVR scenes using CFGs and spatial reasoning rules.The generated dataset is called SPARTQA-AUTO.Source: §4 · p. 4
  • Randomly selecting blocks and objects yields multiple stories per image and spatial reasoning opportunities with missing information.Relationships between blocks are generated randomly, while relationships between objects use the images’ ground truth.Source: §4 · p. 4
  • CFGs verbalize scene relationships in varied forms, including relative and nested relations between objects.Context-sensitive rules constrain grammar options based on selected entities and previously described sentences.Source: §4 · p. 4, Figure 3
  • Four modules manage question consistency and reasoning depth, while FR, YN, FB, and CO use distinct relation-query, quantifier, indirect-reference, and candidate-checking patterns.Questions are generated from stored entity and relationship data rather than by processing raw text.Source: §4 · p. 5
  • Find-all-relations exhaustively infers links using spatial rules, and answers use direct or inferred relations; YN returns DK when neither the queried relation nor its inverse is found.The rules include transitivity, symmetry, converse, inclusion, and exclusion.Source: §4 · p. 5

4.1 Corpus Statistics

  • 9 sentences and 118 tokens per story on average; questions average 23 tokens, with train, dev, and test splits following NLVR image splits.Stories range from 3 to 22 sentences and 66 to 274 tokens; questions range from 6 to 57 tokens.Source: §4.1 · pp. 5–6

5 Models for Spatial Reasoning over Language

The paper casts all four question types as sequence classification for transformer language models, with inputs and inference procedures varying by question type. Training sums classifier cross-entropy losses, and inference includes postprocessing for inconsistent answers.

5 Models for Spatial Reasoning over Language

  • All four question types are sequence-classification tasks that BERT, ALBERT, and XLNet handle using the representation of a prepended [CLS] token.Input sequences and inference vary by question type.Source: §5 · p. 6
  • FR and YN concatenate the story and question; FR classifies each candidate independently and unions answers, while YN selects the highest-confidence label.FR may have multiple correct answers; YN chooses among Yes, No, and DK.Source: §5 · p. 6
  • FB and CO append each candidate answer to the input, then classify the last LSTM output vector across candidates.The LSTM layer was added because the defined YN and FR model was moderately less accurate on FB and CO.Source: §5 · p. 6

5.1 Training and Inference

  • Training sums cross-entropy losses across binary classifiers; inference removes inconsistent FR and YN answers and selects the highest-probability valid YN candidate.FR and YN use multiple classifiers, while FB and CO use one classifier.Source: §5.1 · p. 6

6 Experiments

Experiments test further-pretrained language models on SPARTQA-HUMAN, SPARTQA-AUTO, and external QA datasets. BERT further pretrained on SPARTQA-AUTO improves over original BERT on SPARTQA-HUMAN and performs better with less bAbI training data and on boolQ, while vocabulary shifts and human-test results expose remaining difficulty.

6.1 Further pretraining on SPARTQA-AUTO improves spatial reasoning

  • System 5, BERT further pretrained on SPARTQA-AUTO as a QA task, consistently outperforms original BERT on SPARTQA-HUMAN, though its YN accuracy trails System 3.All systems remain behind human performance by a large margin.Source: §6.1 · pp. 7–8, Table 2 · p. 7
  • Stories-only MLM modestly improves over original BERT, but System 5’s gap over it indicates the annotations provide more supervision than additional unannotated text; QA-task pretraining also beats MLM on stories and QA text.System 4 converts QA pairs into textual statements for MLM pretraining.Source: §6.1 · p. 7, Figure 5 · p. 7
  • On YN accuracy, System 5 is below System 3 and all systems fall below the majority baseline; on YN F1, however, all systems beat that baseline.The authors hypothesize that imbalanced data explains the accuracy pattern.Source: §6.1 · p. 7, Table 3 · p. 7

6 Experiments

  • Experiments further pretrain BERT, ALBERT, and XLNet on SPARTQA-AUTO, then evaluate them on SPARTQA-HUMAN and Seen and vocabulary-modified Unseen sets from SPARTQA-AUTO.Human evaluation fine-tunes the models on SPARTQA-HUMAN training data after further pretraining.Source: §6 · p. 6, §6.2 · p. 8, Table 4 · p. 8

6.2 SPARTQA is challenging

  • Models perform worst on YN, and drops on Unseen and human tests suggest vocabulary overfitting; high Seen scores also appear likely to reflect overfitting.Unseen changes vocabulary without changing reasoning steps; the authors suggest additional quantifiers may make YN more complex.Source: §6.2 · p. 8
  • Consistency and contrast tests assess reliability only on correctly predicted Seen questions, checking whether models preserve answers to equivalent questions and revise them after answer-changing edits.The contrast questions require returning to the story for a new answer.Source: §6.2 · p. 8, Table 5 · p. 9

6.3 Extrinsic evaluation

  • At 99% accuracy on bAbI, BERT (SPARTQA-AUTO) requires 1k training examples, while original BERT requires twice as much.Both reach perfect accuracy with 5k or 10k examples, and the further-pretrained model converges faster in these experiments.Source: §6.3 · pp. 8–9, Figure 6 · p. 9
  • BERT (SPARTQA-AUTO) performs better than original BERT and the reported distant-supervision alternatives on boolQ’s development set.Many correctly answered boolQ examples require multi-step reasoning, which the authors hypothesize is improved by training on SPARTQA-AUTO.Source: §6.3 · p. 9, Table 6 · p. 9

7 Conclusion

The conclusion presents SPARTQA and automatic generation with grammar and spatial reasoning rules as approaches to spatial QA, and reports improved spatial reasoning and extrinsic results. The appendix describes question-template choices and label statistics.

7 Conclusion

  • Further pretraining on automatically generated spatial descriptions and QA annotations significantly enhances LMs’ spatial understanding and reasoning, with better results on bAbI and boolQ.The paper also reports that pretrained LMs perform poorly on the limited human-annotated training data, while humans solve those questions reliably.Source: §7 · p. 9
  • The paper proposes a human-created spatial QA benchmark and uses hand-crafted grammar and spatial reasoning rules to generate descriptions and QA annotations automatically.Source: §7 · p. 9

A Question Templates and statistics Information

  • Question templates use <object> and <relation> variables, filled from story objects and relations using designated modules.Source: A Question Templates and statistics… · p. 13
  • Template articles and quantifiers distinguish question objectives; the generation process checks object-definition uniqueness and varies “any” and “all” in yes/no questions.For example, “Are all blue circles near to a triangle?” differs from “Are there any blue circles near to a triangle?”.Source: A Question Templates and statistics… · p. 13
  • Table 8 reports correct-label percentages for train and test sets, and multi-choice question types can have more than one true label.Source: A Question Templates and statistics… · p. 13

B Sentences of the Dataset

The appendix describes generated sentences with features intended to challenge models and provides detail on three additional evaluation sets.

B Sentences of the Dataset

  • Generated SPARTQA-AUTO sentences include features that challenge models to understand different forms of spatial-relation descriptions.Source: B Sentences of the Dataset · p. 13

C Additional Evaluation Sets

C.1 Unseen Evaluation Set

The unseen test set is designed to check whether models rely on surface-language shortcuts by changing vocabulary while preserving the intended difficulty for consistent models.

C.1 Unseen Evaluation Set

  • The unseen SPARTQA-AUTO test set changes vocabulary in stories and questions to test for shortcuts based on surface language.Random modifications include shape, color, size, and relation names or expressions; they are intended not to affect a consistent and reliable model’s performance.Source: §C.1 · p. 13

C.2 Contrast and Consistency Evaluation

The paper supplements SPARTQA-AUTO with evaluation sets for consistency and contrast, scene-graph annotations, and sentence- and question-level SpRL annotations. It also describes a model fine-tuning architecture.

C.2 Contrast and Consistency Evaluation

  • Consistency questions preserve the information sought through question changes, while contrast questions change the answer; together they probe semantic sensitivity and robustness.Consistency evaluation tests understanding of underlying semantics rather than question-structure overfitting; contrast evaluation checks sensitivity to semantic changes under minor language edits.Source: §C.2 · p. 13

D Extra Annotations

D.1 Detailed Annotation and Scene-Graphs

  • Fine-grained scene graphs represent blocks, relations, and object attributes, including direct relations, and can support QA supervision and evaluation of reasoning steps.They are generated from the story and make spatial relations stated in the text directly available to models.Source: §D.1 · pp. 13–14

D.2 SpRL Annotation

  • Sentence- and question-level annotations use hand-crafted rules and the SpRL scheme to identify spatial expressions and their arguments.The annotations are intended for applications that detect and reason about those expressions and arguments.Source: §D.2 · p. 14

E QA Language Models for Spatial Reasoning over Text

F bAbI and boolQ Datasets

The section reports bAbI task 17 performance for rule-based and BERT models, then describes applying the BERT fine-tuning setting to boolQ.

F bAbI and boolQ Datasets

  • The illustrated bAbI task 17 story gives two vertical relations and asks yes/no questions about the pink, red, and blue squares.Source: F bAbI and boolQ Datasets · p. 14
  • 100% accuracy: the SpRL+rule-based model solves bAbI task 17 by extracting spatial relation triplets and applying spatial rules.It checks whether the asked relation appears among the relations found between the two objects.Source: F bAbI and boolQ Datasets · p. 15
  • 100% accuracy on 10k and 5k training questions, and 99% on 2k and 1k training samples, for the BERT model.The model is fine-tuned with a boolean classifier for Yes/No answers.Source: F bAbI and boolQ Datasets · p. 15
  • For boolQ, the authors further fine-tune BERT using the same setting as the neural network model on bAbI.Source: F bAbI and boolQ Datasets · p. 15
Loading 2104.05832v1…