Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation

1 Introduction

• The paper motivates direct evaluation because practical NLU systems should have commonsense reasoning, while end-to-end systems can struggle with basic plausibility.For example, distinguishing putting a turkey from putting an elephant into a fridge requires commonsense reasoning.Source: §1 · p. 1
• The benchmark directly tests whether systems can distinguish sensible from nonsensical statements and identify the key reason a statement fails.The two tasks use similarly worded statements and a choice of possible explanations.Source: §1 · pp. 1–2
• Human performance is 99.1% on Sen-Making and 97.3% on Explanation across 2,021 instances per task.Version 1.0 was manually labeled by seven annotators.Source: §1 · p. 2
• Models evaluated on the benchmark remain far behind human performance, and detailed examination identifies inference as a challenge.The models were trained on more than 100 million natural-language sentences.Source: §1 · p. 2
• The authors present the dataset as a direct linguistic sense-making benchmark and the first to ask for reasons behind decisions.They hope it will inform interpretability research and combining explanation with language generation.Source: §1 · p. 2

2 Related Work

• The Winograd Schema Challenge requires commonsense knowledge but evaluates it indirectly and does not ask why one option is correct.Its example asks whether the trophy or suitcase was too big to fit.Source: §2 · p. 2
• COPA, JOCI, and SWAG emphasize causes, plausibility, or likely next events rather than reasons events occur or follow.COPA asks users to choose a cause or result; JOCI rates plausibility, and SWAG selects a likely alternative.Source: §2 · p. 2
• Reading-comprehension datasets ask questions about texts, while the paper's benchmark permits external resources that help with its tasks.The cited reading-comprehension examples include Story Cloze, ROCStories, and MCScript.Source: §2 · p. 2
• Earlier benchmark tasks do not directly estimate common sense or ask for the logical reasons behind correct answers.The paper contrasts its benchmark with tasks spanning several commonsense and question-answering approaches.Source: §2 · p. 3
• ConceptNet is a relation-labeled commonsense knowledge graph with associated sentences, and the authors consider it for model fine-tuning.The paper distinguishes such resources from benchmarks that directly evaluate its tasks.Source: §2 · p. 3

3 Testset

• Each instance pairs two similar, syntactically matched statements that differ by a few words, then asks for the valid one and the reason the invalid one fails.Sen-Making selects the valid statement; Explanation selects the sole correct reason from three options.Source: §3 · p. 3
• Version 1.0 contains 2,021 samples, with 2,021 correct and incorrect statements for Sen-Making and one true plus two confusing reasons per sample for Explanation.The authors plan to release a larger dataset with more tasks and samples.Source: §3 · p. 3
• Annotation guidelines favor everyday knowledge, context-related confusing reasons, and alternatives that are valid on their own and similar in length.These are soft restrictions, so some samples do not follow every principle.Source: §3 · p. 3
• Annotators write samples for review, may use external sources for inspiration, and are cautioned not to use the three reason options in Sen-Making.The authors warn that including those options would make the task artificially easy.Source: §3 · p. 4

3.1 Corpus Analysis

• The authors filter differing words to those appearing more than nine times and remove stopwords; negative words occur 53 times in incorrect statements and 55 times in correct ones.Figure 2 displays the different-word analysis; the authors conclude that negative words are not used to construct either class.Source: §3.1 · p. 4, Figure 2 · p. 4

4 Experiments

• For Sen-Making, the experiments score both statements and choose the lower-scoring one; for Explanation, they score statement–reason concatenations.The baselines are state-of-the-art language models trained on large texts, under the assumption that common-sense knowledge is encoded in text.Source: §4 · p. 4
• Human evaluation uses at least three testees per sample, rewriting or removing samples that more than half answer incorrectly.Other samples are retained and recorded in the results.Source: §4 · p. 4

4.1 Results & Analysis

• ELMo and BERT outperform random results on Sen-Making, but ELMo is near-random on Explanation while BERT beats random there.ELMo performs better than BERT on Sen-Making; BERT performs better than ELMo on Explanation.Source: §4.1 · p. 5, Table 1 · p. 5
• Fine-tuning improves ELMo's Sen-Making, barely changes its Explanation, and leaves BERT's Sen-Making unchanged while worsening its Explanation.The authors suggest added knowledge may help identify common sense without helping inference, while fine-tuning may disrupt BERT's Next Sentence Prediction structure.Source: §4.1 · p. 5
• A thorough investigation of fine-tuning human-labeled knowledge bases is beyond the paper's scope, and those bases are limited in scope.The authors consider gleaning knowledge from large raw text a more promising direction.Source: §4.1 · p. 5
• Human performance remains near 100% on sense making, and most human errors reflect reduced concentration rather than conceptual issues.The authors contrast this with reading comprehension tasks where careful fine-tuning can let machines surpass humans.Source: §4.1 · p. 5

4.2 Case Study

• On the New York example, fine-tuned ELMo identifies the incorrect statement after receiving city and country facts but still cannot select its reason.The original ELMo fails both the sense-making and explanation decisions in this example.Source: §4.2 · p. 5
• Language-model techniques fail on the multi-step lunch inference example whether fine-tuned or not, and the explanation remains difficult too.The authors suggest multi-step inference remains difficult for LSTM language models.Source: §4.2 · p. 5

• The benchmark directly evaluates sense making and explanation; results indicate that sense making remains challenging for models and inference is a key missing factor.Source: §5 · p. 5