Source-linked AI summary
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton
TL;DR
NLU systems have achieved strong benchmark performance, but their systematic and robust generalization remains difficult to isolate and assess. CLUTRR addresses this gap with kinship stories, held-out logical-rule combinations, and curated noise facts, and finds that symbolic graph reasoning outperforms existing text-based NLU systems on generalization and robustness.
Problem
NLU benchmarks make it difficult to diagnose systematic generalization and robustness because they combine reasoning with parsing, entity resolution, and world knowledge.
Method
CLUTRR uses semi-synthetic family stories requiring models to extract relationships, induce kinship rules, and infer unstated relations under held-out rule combinations and added noise facts.
Results
Existing NLU systems exhibit poorer generalization and robustness than a Graph Attention Network given direct access to symbolic story representations.
Takeaways & Limitations
CLUTRR exposes a remaining gap between reasoning from unstructured text and reasoning from structured input, supporting its use as a diagnostic benchmark.
Takeaways & Limitations
Kinship and gender roles are oversimplified to maintain tractability, and clauses of length k = 2 are not held out so models can learn all binary-predicate compositions.
Abstract
from arXiv · showhide
The recent success of natural language understanding (NLU) systems has been troubled by results highlighting the failure of these models to generalize in a systematic and robust way. In this work, we introduce a diagnostic benchmark suite, named CLUTRR, to clarify some key issues related to the robustness and systematicity of NLU systems. Motivated by classic work on inductive logic programming, CLUTRR requires that an NLU system infer kinship relations between characters in short stories. Successful performance on this task requires both extracting relationships between entities, as well as inferring the logical rules governing these relationships. CLUTRR allows us to precisely measure a model's ability for systematic generalization by evaluating on held-out combinations of logical rules, and it allows us to evaluate a model's robustness by adding curated noise facts. Our empirical results highlight a substantial performance gap between state-of-the-art NLU models (e.g., BERT and MAC) and a graph neural network model that works directly with symbolic inputs---with the graph-based model exhibiting both stronger generalization and greater robustness.
1 Introduction
CLUTRR addresses concerns that NLU systems may rely on dataset artifacts rather than systematic, robust reasoning. It isolates these abilities through kinship stories, held-out rule combinations, and noise facts, revealing a gap between text-based NLU systems and a symbolic graph model.
- Motivation: Existing NLU benchmarks often entangle systematic generalization with co-reference resolution, world knowledge, and semantic parsing.This makes it difficult to isolate and diagnose robustness and systematicity.
- Benchmark: CLUTRR is a semi-synthetic benchmark designed to test systematic and robust logical generalization in NLU models.It is inspired by inductive logic programming and visual reasoning benchmarks.
- Benchmark: In CLUTRR, models infer an unstated kinship relation from a short hypothetical-family story.Solving the task requires extracting textual relationships, inducing kinship rules, and applying them to entity pairs.
- Evaluation: Held-out combinations of logical rules measure systematic generalization, while superfluous noise facts test model robustness.These controls allow the benchmark to distinguish rule composition from resistance to irrelevant information.
- Results: Existing NLU systems show substantially weaker generalization than a Graph Attention Network given symbolic story representations.Training on noisy data generally does not improve NLU robustness, while the GAT learns robust reasoning strategies from noisy examples.
2 Related Work
CLUTRR builds on reading-comprehension, inductive-logic-programming, and language–knowledge-graph research while targeting a distinct diagnostic setting. Unlike prior knowledge-graph benchmarks, it evaluates inductive reasoning over previously unseen entities.
- Foundations: The benchmark draws inspiration from inductive logic programming and prior work combining language with knowledge graphs.Its kinship-reasoning setup follows a classic family-relation induction task.
- Reading comprehension benchmarks: CLUTRR is positioned within reading-comprehension benchmarks that test factual question answering, sentence understanding, and related reasoning abilities.The paper’s primary contribution is a diagnostic benchmark for robustness and systematic generalization in NLU.
- Knowledge-graph reasoning: Knowledge-graph question-answering systems either receive a graph for inference or infer one from text.These tasks provide context for CLUTRR’s text-to-structure and reasoning objectives.
- Knowledge-graph reasoning: Unlike generally transductive prior benchmarks, CLUTRR requires inductive logical reasoning over a new set of previously unseen entities in every example.This shifts evaluation from fixed-entity knowledge use toward compositional reasoning across novel instances.
3 Benchmark Design
CLUTRR is a semi-synthetic benchmark for testing inductive reasoning, systematic generalization, and robustness in natural-language kinship inference. It generates controlled reasoning problems by combining logical kinship graphs, backward-chained facts, and diverse narrative realizations, including curated noise.
- Benchmark motivation: CLUTRR adapts the classic inductive logic programming task of inferring unstated kinship relations from stated family facts.The benchmark requires models to compose induced logical rules, such as inferring a grandfather relation from parent relations.
- Data generation: CLUTRR generates each example by constructing a kinship graph, sampling a target relation, backward-chaining supporting facts, and converting them into a natural-language story.The generation process uses a knowledge base of kinship rules and can optionally add irrelevant or distracting facts.
- Diagnostic variations: CLUTRR supports systematic-generalization and robustness diagnostics through controlled variations, including held-out rule combinations and categorized noise paths.Its released suite includes generation code, datasets, and model baselines; noise can be irrelevant or supporting relative to the reasoning path.
- Benchmark motivation: The benchmark evaluates natural-language reasoning by asking models to infer the relationship between two entities whose relationship is not explicitly stated.Solving the task requires extracting textual relationships, inducing kinship rules, and applying those rules to the queried pair.
- Data generation: The generated stories are grounded in logical clauses whose target fact is supported by a set of facts sampled from the kinship graph.The knowledge base contains general rules, whereas each graph contains grounded relations for one story; rules are recursively applied to complete the graph.
- Adding natural language: Crowd-sourced paraphrasing converts supporting facts into varied narratives by allowing workers to split and combine facts across sentences.CLUTRR also programmatically recombines collected natural-language templates to generate stories with different properties.
4 Experiments
The experiments evaluate CLUTRR models on systematic generalization, structured reasoning, and robustness to noisy facts. Across these settings, the graph-based GAT generally outperforms text-based NLU models, although each model class shows distinct sensitivities.
- Experimental setup: The experiments compare text-based NLU baselines, including BiLSTMs, Relation Networks, MAC, and BERT, with a GAT receiving structured graph representations.All models encode the story or graph and predict the target relation from the resulting representation and target entities.
- Q1: Systematic generalization: Models are evaluated on clauses requiring 2–10 reasoning steps, with training regimes using clause lengths k = 2, 3 or k = 2, 3, 4.Datasets contain 5000 training stories per k value and 100 test stories per k value; results average 10 random runs.
- Q1: Systematic generalization: The GAT performs near-perfectly on held-out clauses of length k = 3, while BERT-LSTM is the strongest text-based model but remains significantly below GAT.Performance degrades monotonically as test-clause length increases, and all models improve when trained with k = 2, 3, 4 rather than k = 2, 3.
- Q2: The Benefit of Structure: The GAT’s stronger systematic-generalization performance suggests that text-based models fail to learn a robust mapping from narratives to the underlying logical facts.The comparison directly contrasts models processing natural-language narratives with a model given the underlying logical graph.
- Q3: Robust Reasoning: The GAT outperforms unstructured text-based models across most noise-testing scenarios, highlighting the benefit of structured feature spaces for robust reasoning.When trained on noisy examples, only GAT consistently improves its performance.
- Q3: Robust Reasoning: Adding supporting facts harms GAT performance, whereas disconnected facts cause no performance drop, suggesting sensitivity to cycles but robustness to disconnected noise.Most text-based models excluding BERT can improve when supporting or irrelevant facts are added, while BERT-based models do not benefit from the extra content.
5 Conclusion
The paper concludes that CLUTRR diagnoses systematic generalization and inductive reasoning in NLU systems. Existing text-based systems show poorer robustness and generalization than a graph neural network operating on symbolic input, motivating more compositional and modular NLU systems.
- CLUTRR is introduced as a benchmark for testing systematic generalization and inductive reasoning capabilities in NLU systems.
- Existing NLU systems exhibit relatively poor robustness and systematic generalization, especially compared with a graph neural network using symbolic input.
- The results identify a remaining gap between reasoning models operating on unstructured text and models given more structured input.The authors hope CLUTRR supports progress toward more compositional, modular, and robust NLU systems.
1 Appendix
The appendix examines robustness and systematic generalization under altered placeholders, noisy facts, embedding policies, and human evaluation. Results show that language-model performance depends strongly on placeholder design, while GAT behavior varies with noise structure and training regime.
- Placeholder size and split: Using 20% of the placeholders without a train-test placeholder split makes all language-based models competitive with GAT for both k = 2, 3 and k = 2, 3, 4.The result indicates that separating placeholder sets is necessary to test systematic generalization effectively.
- Robust reasoning: Training GAT with noisy data improves performance across testing scenarios, but cycles and dangling paths create contrasting attention and generalization effects.Cycle-trained GAT attends to all paths, which harms evaluation on irrelevant facts; training on irrelevant facts enables attention to only relevant paths.
- Synthetic placeholders: Toy placeholders markedly increase NLU-model performance, sometimes allowing them to outperform GAT, highlighting the importance of paraphrased placeholders for dataset complexity.The experiments use bABI-style simple placeholders to isolate the effect of language variation.
- Entity embeddings: Fixed random entity embeddings produce slightly better systematic-generalization scores for Bidirectional LSTM, while embedding-policy differences have practically no effect on GAT.The reported experiments use fixed random embeddings.
- AMT data collection: Peer review accepted 79% of collected paraphrases when all reviewers agreed on their quality, and this subset was used in the benchmark.The review process aimed to remove logical and grammatical inconsistencies.
- Human evaluation: Experts solved puzzles accurately with ample time, whereas time pressure caused missed relational details; difficulty ratings increased with task complexity.Experts took an average of 6 minutes per puzzle, while Turkers averaged 1 minute 23 seconds.
2 Supplemental Material
The supplemental material documents reproducibility information, model hyperparameters, dataset snapshots, and the data-collection interface. It also records the released data and the organization of evaluation resources.
- Reproducibility: The experiments use 10,000 training examples and 100 testing examples for each testing scenario, with an 80-20 random training-dev split on each run.These details are reported as part of the publicly released experiment information.
- Hyperparameters: The common experimental settings use Adam with learning rate 0.001, 100 epochs, and 10 runs, alongside 100-dimensional embeddings except for BERT-based models.Model-specific hyperparameters are listed separately for each architecture.
- Data collection: Figure 8 presents the Amazon Mechanical Turk interface used for data collection and peer reviews.The interface was built using ParlAI.