A Hybrid Neural Network Model for Commonsense Reasoning

1 Introduction

• Pronoun resolution can require commonsense knowledge about who typically fears or advocates violence.The city-councilmen examples contrast the pronoun referent according to whether the subject feared or advocated violence.Source: §1 · p. 1
• Resolving “it” can depend on whether the object is too big or the container is too small.The trophy-and-suitcase example illustrates how commonsense knowledge helps determine the pronoun's referent.Source: §1 · p. 1
• WSC and PDP are studied as classic commonsense reasoning tasks formulated as anaphora resolution.The passage describes anaphora resolution as a form of commonsense reasoning.Source: §1 · p. 1

A. The trophy

• The WSC examples ask readers to identify the reference of a pronoun, including a case where “it” refers to what is too small.The table caption says the pronoun to resolve is bold.Source: Table 1 · p. 1, A. The trophy · p. 1
• WSC and PDP are described as better approximations of human reasoning than other listed tasks, while remaining challenging for machines.The passage also characterizes WNLI as GLUE's most challenging NLU task and says most models can hardly outperform the majority-voting baseline scored at 65.1.Source: A. The trophy · pp. 1–2
• Traditional commonsense methods rely heavily on human-crafted features and knowledge bases, while the paper explores deep neural networks.The related-work discussion distinguishes language models that score substituted statements from semantic similarity models that compare pronouns and candidate antecedents.Source: A. The trophy · p. 2
• HNN combines a masked language model and a deep semantic similarity model that share BERT but use different model-specific input and output layers.The final score combines the two model scores; training first pretrains the text encoder on raw text, then applies multi-task learning on WSCR.Source: A. The trophy · p. 2
• WNLI reaches 89%, WSC reaches 75.1%, and PDP-60 reaches 90.0% as HNN's new state-of-the-art results.The paper also reports an ablation study to justify HNN's design.Source: A. The trophy · p. 2

2.1 Masked Language Model (MLM)

• MLM replaces the pronoun with N [MASK] tokens, where N matches candidate length, then scores candidate likelihood and averages token log-probabilities for multi-token candidates.Source: §2.1 · p. 3

2.2 Semantic Similarity Model (SSM)

• SSM uses the contextual [CLS] embedding for the sentence and attention over candidate-token embeddings to construct the candidate representation.The input pairs sentence S with candidate C, separated by [SEP].Source: §2.2 · p. 3
• SSM represents the pronoun with its first-token contextual embedding, scores semantic similarity bilinearly, and applies a logistic function to predict whether the candidate is correct.Source: §2.2 · pp. 3–4

2.3 The Training Procedure

• Training uses 1,886 WSCR sentences paired with positive and negative antecedents, initializes BERT from the uncased large model, and optimizes MLM, SSM, and pairwise ranking losses.The ranking objective seeks a larger combined score for the correct candidate than for the incorrect one.Source: §2.3 · p. 4

3.1 Datasets

• WSCR supplies training data for WSC and PDP60, while WNLI's official training and development instances are merged into its final development set.WSCR contains 1,886 instances, including 1,322 for training; WNLI's merged set contains 634 training and 71 development instances.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 Implementation Detail

• WNLI instances are converted to WSC format by detecting premise pronouns and extracting candidates through matching hypothesis substrings.The shared layer is initialized from BERT uncased large; inputs are tokenized with WordPieces and capped at 512 tokens.Source: §3.1 · p. 4, §3.2 · pp. 4–5, Table 3 · p. 6

3.3 Results

• 83.6% vs 71.9% on WNLI and 75.1% vs 72.2% on WSC accuracy: HNN surpasses previous bests and achieves new state-of-the-art results across three benchmarks.The comparisons are against BERTWiki-WSCR on WNLI and WSC.Source: §3.3 · p. 5, Table 4
• 89.0% vs 83.6% accuracy on WNLI: majority voting over the last six epoch predictions yields HNNensemble, a 5.4% absolute gain.The final prediction combines the last six model predictions.Source: §3.3 · p. 6, Table 4

3.4 Ablation study

• Removing SSM reduces WNLI from 77.1% to 74.5%; removing MLM leaves 75.1%, supporting component complementarity.Table 5 reports WNLI and WSCR on development sets and WSC and PDP60 on test sets.Source: Table 5 · p. 6, §3.4 · p. 6
• Without ranking loss, performance drops on WNLI, WSCR, and WSC; PDP60 is slightly better, but its difference is not statistically significant.The authors therefore retain ranking loss in HNN's training objective.Source: Table 6 · p. 7, §3.4 · p. 7

Is the WNLI task a ranking or classification task?

• 77.1% vs 74.6% accuracy for HNN: ranking beats classification on WNLI, and the same direction holds for MLM and SSM.For classification, the model applies a threshold; for ranking, it selects the top-ranked candidate.Source: Figure 3 · p. 7, Is the WNLI task a ranking or… · p. 7

• HNN combines a masked language model and a deep semantic similarity model that share a BERT-based contextual encoder but use different input and output layers.Source: §4 · p. 7
• 89% on WNLI, 75.1% on WSC, and 90.0% on PDP60 are reported as new state-of-the-art results for HNN.Source: §4 · p. 7
• Future work will extend HNN to more sophisticated reasoning tasks, especially those where large-scale language models like BERT and GPT do not perform well.Source: §4 · p. 7