Ordinal Common-sense Inference

1 Introduction

• Common-sense inferences are expected in context but fall short of logical entailment, while corpus reporting bias limits what frequency alone can reveal about real-world likelihood.The paper characterizes these inferences as possibilistic.Source: §1 · p. 1
• Unstated real-world facts can have likelihoods that do not align with language-model probabilities, motivating systems that assess inferences in context.The target is distinct from predicting sentences likely to be reported in the same text.Source: §1 · p. 2
• The proposed knowledge-acquisition process derives possible statements from text, then uses human annotation to assess their likelihood in grounded contexts.The ordinal task covers judgments ranging from very likely to impossible.Source: §1 · p. 2
• The paper combines large-scale extraction and abstraction, crowdsourced assessment, and neural forward generation to build the JOCI corpus and report baseline prediction results.JOCI contains diverse common-sense inference examples judged to have varying subjective likelihood.Source: §1 · p. 2

Mining Common Sense

• Common-sense knowledge mining ranges from costly expert work and volunteer efforts to automated extraction methods, including abstraction from descriptive text.The paper uses mined knowledge to seed inference-example generation.Source: §2 · p. 2

Common-sense Tasks

• Crowdsourced elicitation can be scalable but may miss plausible inference pairs that people can judge when examples are supplied.The paper instead relies on human judgments of provided examples rather than asking people to generate them.Source: §2 · pp. 2–3

Plausibility

• Ordinal plausibility is graded like textual similarity but conditions a hypothesis on its preceding context rather than measuring unconditional sentence plausibility.Annotators explicitly judge plausibility on a five-point ordinal scale.Source: §2 · pp. 3–4

Textual Entailment

• Strict textual entailment differs from common-sense likelihood: a hypothesis can be plausible without being entailed, as with a coin flip coming up heads.The paper distinguishes an entailment probability from the likelihood that an inference is true.Source: §2 · pp. 3–4

Non-entailing Inference

• Narrative-cloze methods predict what appears in a document, whereas this work targets common-sense inferences that are often true but never stated there.The paper also considers a prior five-point Likert evaluation focused on generating very likely inferences, rather than a wide range including impossible ones.Source: §2 · p. 4

3 Ordinal Common-sense Inference

• The task uses five likelihood labels: very likely, likely, plausible, technically possible, and impossible.The scale is framed as subjective likelihood on an ordinal scale.Source: §3 · pp. 4–5, Figure 2 · p. 5
• Annotators assess whether a provided hypothesis holds in a context, extending entailment judgments to graded epistemic modality.The scale builds on Horn’s modality values and refines “possible” into plausible and technically possible.Source: §3 · p. 4
• The task generates hypotheses automatically, which the authors say seeks to avoid the elicitation bias of prior work.Source: §3 · p. 5

4 Framework for collecting OCI corpus

• The framework has two stages: automatically generate inference candidates from contexts, then annotate them with ordinal labels.The annotation stage asks humans to judge candidate hypotheses in their context.Source: §4 · p. 5

4.1.1 Generation based on World Knowledge

• World-knowledge generation extracts propositions from text, abstracts them into templates, derives concept properties, and turns those properties into hypotheses grounded in a context.The workflow uses Gigaword propositions and WordNet senses and properties.Source: §4.1.1 · pp. 5–6

4.1.2 Generation via Neural Methods

• Neural candidate generation trains a sequence-to-sequence model on SNLI entailment pairs, mapping premises to hypotheses.It uses either a full context sentence or a single argument word as input; the single-word strategy is intended to encourage more general hypotheses.Source: §4.1.2 · pp. 6–7

Neural Sequence-to-Sequence Model

• The sequence-to-sequence model estimates a hypothesis given a context using an encoder-decoder architecture, attention, and autoregressive word prediction.The encoder’s final state initializes the decoder, while attention combines encoder states for output prediction.Source: Neural Sequence-to-Sequence Model · p. 7

4.2 Ordinal Label Annotation

• Annotators rate each context-hypothesis pair on five ordinal labels, from very likely to impossible, and can mark nonsensical or ungrammatical candidates NA for filtering.The five labels correspond to ordinal values 5 through 1.Source: §4.2 · pp. 7–8

5 JOCI Corpus

• JOCI combines generated inferences grounded in SNLI and ROCStories contexts with ordinally annotated pairs drawn directly from SNLI and other corpora.The direct pairs are re-annotated under the same protocol.Source: §5 · p. 8

5.1 Data sources for Context-Hypothesis Pairs

• The main corpus uses first sentences from SNLI image-caption pairs and ROCStories, with generated candidates collected across SNLI splits and sampled ROCStories contexts.For SNLI training data, the authors randomly selected 4,600 arguments and one sentence containing each argument.Source: §5.1 · p. 8
• Existing-corpus pairs include SNLI premises with entailment, neutral, or contradiction hypotheses, ROCStories first sentences paired with later sentences, and COPA premises paired with effects.The SNLI pairs were sampled from the training set; ROCStories uses its second or third sentence as the hypothesis.Source: §5.1 · p. 8

5.2 Crowdsourced Ordinal Label Annotation

• Crowdsourced annotators judge hypothesis plausibility during or shortly after the context; about 25% marked nonsensical are removed, and three workers' labels are reduced to a median.Workers could skip ordinal annotation for hypotheses that did not make sense; this category includes incomplete or grammatically incorrect sentences.Source: §5.2 · pp. 8–9

5.3 Corpus Characteristics

• Average Cohen’s κ is 0.54, and filtering for averaged κ above 0.6 still leaves over 20K context-hypothesis pairs.Pilots used a 0.7 average-κ threshold to qualify a pool of over 150 workers.Source: §5.3 · p. 9, §5.2 · p. 9
• JOCI covers a wide range of ordinal likelihood labels, while SNLI entailments are mostly judged very likely, neutral pairs plausible, and contradictions impossible or technically possible.JOCI also has many samples at each ordinal scale.Source: §5.3 · p. 10, Figure 6 · p. 10
• In COPA’s plausible-effect pairs, most alternatives are plausible or likely, and the preferred COPA-1 alternative receives a higher likelihood judgment than COPA-0 in almost every pair.The authors include effects rather than causes; Figure 8 displays the joint ordinal-label distribution for the paired alternatives.Source: §5.3 · p. 10, Figure 8 · p. 11
• Full-sentence prompts yield more very-likely labels than single-word prompts, while the ISA world-knowledge strategy produces a bimodal distribution concentrated at impossible or very likely.The authors suggest that full-sentence prompts may make the neural model behave more like it does on SNLI entailments.Source: §5.3 · pp. 10–11

6 Predicting Ordinal Judgments

• The ordinal regression model uses margin-based training and features defined over context-inference pairs.The feature list itself is not included in the supplied passages.Source: §6 · p. 11

6.1 Analysis

• A contains 2,976 SNLI-train pairs with human-elicited hypotheses; B contains 6,375 pairs using the same premises and automatically generated hypotheses.Both subsets are annotated with ordinal labels.Source: §6.1 · p. 12
• The regression model achieves the lowest MSE and highest ρ; A-test performs better than B-test, which the authors consider more challenging because auto-generated hypotheses vary more.Tables 4 and 5 report MSE and Spearman’s ρ; the supplied passages do not give their cell values.Source: §6.1 · p. 12, Table 4 · p. 12, Table 5 · p. 12
• On A-test, removing similarity and bag-of-words features together produces the largest reported performance drop.Source: §6.1 · p. 12, Table 6 · p. 12
• The authors interpret the analysis as showing statistical differences between human-elicited and automatically generated hypotheses.Source: §6.1 · pp. 12–13

• The authors seek to learn both possible common-sense statements and their context-specific likelihoods from corpora, noting that they know of no prior demonstration of feasibility.Source: §7 · p. 13
• The approach extracts possible statements from a corpus, grounds them in contexts, and uses human judgments of subjective likelihood rather than corpus data alone.The authors say automatic example generation seeks to bypass human elicitation bias and that the crowdsourcing technique is inexpensive and scalable.Source: §7 · p. 13
• Future work will extend forward-inference generation, scale annotation, and explore larger, more complex contexts.The authors also plan to use JOCI to improve algorithms for natural-language inference tasks such as storytelling and story understanding.Source: §7 · p. 13