Multi-Step Inference for Reasoning Over Paragraphs

1 Introduction

• ROPES questions can require linking information about pollinators in a situation to background statements about pollinators and flower-petal color while tracking polarity.The example requires using those links to determine the category of flowers.Source: §1 · p. 1, Figure 1 · p. 1
• Prior approaches used symbolic reasoning, which can track logical operators but is brittle to language variability, or neural networks, which are more robust to paraphrase but lack explicit reasoning and are hard to interpret.The cited examples of symbolic methods include Markov logic networks and integer linear programming; neural examples include Jiang et al. (2019) and Jiang and Bansal (2019).Source: §1 · p. 2
• The model is a compositional middle ground between symbolic reasoning and black-box neural networks, chaining free-form predicates and logical connectives.It uses general modules targeted at ROPES reasoning and composes them to answer questions.Source: §1 · p. 2
• SELECT identifies question-relevant spans, CHAIN captures interactions among statements, and PREDICT scores potential answers.The modules can be instantiated separately and freely combined.Source: §1 · p. 2
• 29% and 8% relative error reduction on ROPES dev and test sets, respectively, when multi-step inference is combined with a reranker.The authors also characterize the official dev set as more in-domain and the official test set as more out-of-domain.Source: §1 · p. 2

2.1 Baseline

• The baseline concatenates background, situation, and question inputs, then scores ROBERTA tokens for answer-span starts and ends.The extracted span uses the highest start and end scores.Source: §2.1 · p. 2, §2 · p. 2

2.2 Multi-Step Inference for ROPES

• The multi-step model uses a fixed module combination to select relevant text, chain it across passages, and predict an answer.It selects from the question passage, chains with background information, then selects and chains with situation information before predicting.Source: §2.2 · p. 3
• SELECT summarizes important input parts, CHAIN summarizes interactions between inputs, and PREDICT scores candidate answers.PREDICT scores tokens as start/end probabilities in the base model and gives each reranker span a single score.Source: §2.2 · pp. 3–4

2.3 Multi-Step Reranker

• The reranker applies multi-step inference to the top candidate spans from the baseline rather than scoring every input token.Candidate spans are represented by concatenating endpoint tokens; an ensemble chooses the span with the most votes.Source: §2.3 · p. 4

Bias

• The authors recommend treating the dev set as an in-domain test set and the original test set as out-of-domain.They attribute the distributional shift to the annotator split used to separate train, dev, and test.Source: §3 · p. 4

Answer types

• The analysis classifies an answer by the constituent label of the lowest syntax-tree subtree covering its span.When multiple spans match, the analysis uses the last occurrence.Source: §3 · p. 4
• Most noun-phrase answers come from the situation, while other answer types typically come from the question.The four most frequent types are noun phrase, verb phrase, adjective phrase, and adverb phrase.Source: §3 · pp. 4–5

Bias

• Train and dev answers are 85% noun phrases, while fewer than half of test answers are noun phrases, with more verb, adjective, adverb, and other types.The authors describe this answer-type shift as a challenge for reading-comprehension systems.Source: §3 · p. 5

4.1 Settings

• The experiments use 10,924 training, 1,688 dev, and 1,710 test questions, and report EM because F1 correlates poorly with human judgments on ROPES.They also create a 1,100-question train-dev split from training data to evaluate on an in-domain held-out set.Source: §4.1 · pp. 5–6

4.2 Results

• On dev, the multi-step system and multi-step reranker beat baseline by 8.1% and 11.7% absolute EM; on test, by 2.4% and 2.0% EM, respectively.With the ensemble, the multi-step reranker performs best on both sets.Source: Table 5 · p. 6, §4.2 · p. 6
• The model still gains 7.2% EM on dev-test; the authors attribute the larger dev-versus-test gap to distribution shift rather than dev overfitting.They identify handling test-set distribution shift as a future challenge.Source: §4.2 · p. 6

4.3 Analysis and Discussion

• Removing any module reduces performance; Q SELECT and B CHAIN matter more for the multi-step system, while SELECT and S CHAIN removals hurt the reranker more than removing B CHAIN.The reported drops are around -2.1% EM for the multi-step system’s B SELECT or S CHAIN removals, around -5.9% EM for the reranker’s Q SELECT, B SELECT, or S CHAIN removals, and -3.7% EM for its B CHAIN removal.Source: §4.3 · pp. 6–7, Table 6 · p. 7
• All three systems do better on NP, ADJP, and ADVP questions than VP questions, whose answers tend to be longer and more complex.The authors link the dev-test score gap to dev’s larger NP share and test’s relatively larger VP share.Source: §4.3 · p. 7, Table 7 · p. 7
• Three-turn self-sampling yields training accuracy most similar to baseline dev (59.7% EM) and test (55.4% EM), unlike high-accuracy k-fold samples, so the authors adopt it.The k-fold and k-turn strategies partition training data differently to generate candidates.Source: §4.3 · p. 7
• Top 3 candidates provide a good development-set trade-off: fewer candidates lower the reranker’s oracle upper bound, while more incorrect candidates add noise.The authors use top 3 in their main experiments.Source: §4.3 · p. 7
• Remaining errors include numeric-relation reasoning and difficulty associating relevant information with the correct entities amid question-irrelevant entities.The authors suggest that future work could add modules for the entity-association problem.Source: §4.3 · pp. 7–8

Multi-Hop Reasoning

• Neural Module Networks moved from visual question answering to reading comprehension, where FIND and COMPARE retrieve entities for HotpotQA or DROP; ROPES uses modules for chained inference.Source: §5 · p. 8
• ROPES reasoning links effects across background and situation without retrieving from the background, unlike multi-hop reasoning across documents.Source: §5 · p. 8

Models beyond Pre-trained Transformer

• The authors place their work among task-specific systems built beyond pretrained transformers while adopting pretrained contextualized representations.Source: §5 · p. 9

• The model performs chained inference over natural-language text and substantially outperforms prior work on ROPES.Source: §6 · p. 9
• The paper also presents analysis of ROPES intended to inform future work on the dataset.Source: §6 · p. 9
• Unlike neural module networks, the model uses one fixed layout; future work could combine text modules to support reasoning across multiple datasets.Source: §6 · p. 9