Source-linked AI summary
Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, Peter Clark
TL;DR
Process paragraphs require reasoning about entity-state changes whose causal effects are often implicit, while prior systems that worked well on synthetic data perform only slightly above majority baselines on ProPara. The paper introduces a natural-text dataset and two neural models for state prediction and propagation; the models improve accuracy by up to 19%.
Problem
Process-paragraph comprehension requires inferring entity-state changes across sentences because the causal effects of actions are often implicit.
Source: §1 · p. 1Method
The paper creates ProPara, a natural-text dataset annotated for entity states, and proposes two neural models using LSTM input encoding and span prediction.
Source: §1 · p. 2Results
The new models improve accuracy by up to 19% on ProPara, where prior state-of-the-art systems perform only slightly better than majority baselines.
Source: §1 · p. 2Takeaways & Limitations
ProPara presents challenges for previous models and provides a dataset and models for community use.
Source: §6 · p. 9Forward-propagating ProGlobal, EntNet, and QRNs cannot learn the bias to propagate state information backward in time.
Source: §5.5 · p. 9
Abstract
from arXiv · showhide
We present a new dataset and models for comprehending paragraphs about processes (e.g., photosynthesis), an important genre of text describing a dynamic world. The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints). The end-task, tracking the location and existence of entities through the text, is challenging because the causal effects of actions are often implicit and need to be inferred. We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input encoding and span prediction. The new models improve accuracy by up to 19%. The dataset and models are available to the community at http://data.allenai.org/propara.
1 Introduction
Process-paragraph comprehension requires inference over changing entity states whose causal effects are often implicit. ProPara supplies natural process text with state annotations, and the proposed models improve on prior systems that perform only slightly above majority baselines.
1 Introduction
- Questions about process paragraphs can require tracking unstated state transitions across sentences, such as inferring where sugar is produced from where carbon dioxide enters.The action’s causal effects on the changing world state are often implicit.Source: §1 · p. 1
- Earlier process datasets use machine-generated or simulated language, while synthetic-data approaches can struggle with the complexity of organic, real-world language.bAbI uses machine-generated language in a deterministic domain with a small lexicon; SCoNE uses synthetic domains and assumes a complete initial-state model.Source: §1 · pp. 1–2
- ProPara provides human-authored procedural paragraphs with annotations of entity existence and location during real-world processes.The authors describe it as the first dataset combining annotated natural text about real-world processes with a representation of entity states.Source: §1 · p. 2
- Existing state-of-the-art systems perform only slightly better than majority baselines on ProPara, while the new models improve accuracy by up to 19%.The new models use alternative mechanisms for state prediction and propagation, including LSTM input encoding and span prediction.Source: §1 · p. 2
2 Related Work
Prior procedural-text datasets largely use simulated or machine-generated worlds and address tasks distinct from tracking state changes. Earlier neural state-tracking systems maintain state representations but decode final answers from a vocabulary rather than paragraph spans.
2 Related Work
- bAbI and SCoNE use machine-generated or simulated worlds, and SCoNE predicts end states from a given perfect initial-state model rather than addressing ProPara’s state-change task.SCoNE’s motivation also differs, focusing on ellipsis and coreference in context.Source: §2 · p. 2
- ProRead and related event-structure approaches support questions about event ordering but do not model state changes.SCoNE also differs in task and motivation from ProPara.Source: §2 · p. 2
- EntNet represents world state with dynamic memory blocks and gated updates, while QRN propagates a hidden paragraph vector and reduces the query at each step.NPN updates entity-state vectors from predicted actions and affected entities, but does not model different effects of the same action on different entities.Source: §2 · pp. 2–3
- EntNet and QRN decode final vectors into vocabulary entries, whereas factoid QA systems can predict answer spans from the original paragraph.The paper combines span prediction with state propagation in its new models.Source: §2 · p. 3
3 The ProPara Dataset
ProPara targets scientific process paragraphs and represents participant states in a grid over process steps. The dataset combines crowdsourced paragraphs with expert-reviewed entity and existence annotations and worker-provided locations.
3 The ProPara Dataset
- ProPara asks systems to identify whether and when an entity is created, destroyed, or moved, and where those changes occur.The intended process-comprehension questions include inputs, conversions, and the location of a conversion.Source: §3 · p. 3
- The state grid has one column per participant and one row for the states after each step, with each cell recording that participant’s location.A paragraph with m sentences and n participants yields an (m + 1) × n grid; the initial row records locations before the process.Source: §3 · p. 3
- Workers authored paragraphs from process prompts; short paragraphs were removed, leaving 488 paragraphs describing 183 processes.Each prompt was given to five workers, who independently wrote paragraphs.Source: §3 · p. 3
- MTurk workers filled location cells, which could be “unknown” when unspecified or a span of the paragraph; five grids per paragraph yielded average pairwise agreement of 0.67 across 81,345 annotations.Source: §3 · p. 3
- This work traces participant locations, though the representation can be extended to other properties such as temperature.Source: §3 · p. 3
4 Models
The paper presents ProLocal, which predicts local state changes and propagates them, and ProGlobal, which predicts states using global context and previous-step information.
4 Models
- ProLocal predicts local state changes and propagates them, while ProGlobal makes end-to-end predictions using global information.ProLocal uses algorithmic propagation; ProGlobal models state persistence within the neural model.Source: §4 · p. 4, §4.1 · p. 4
4.1 ProLocal: A Local Prediction Model
- ProLocal combines sentence-level state predictions with a commonsense persistence rule that fills missing locations forward and backward.Locations default to unchanged unless an action specifies otherwise; conflicting information yields an unknown location.Source: §4.1 · p. 4, §4.1.2 · p. 5
4.1.1 Local Prediction
- Local prediction classifies an entity’s change type and predicts before-and-after location spans from the sentence using a BiLSTM with attention.Separate attention weights serve state-change classification and location tagging because different sentence cues may matter for each task.Source: §4.1.1 · pp. 4–5
4.1.2 Commonsense Persistence
- Commonsense persistence propagates known locations through unfilled states, assuming entities remain unchanged unless told otherwise.If locations clash, the predicted location is unknown.Source: §4.1.2 · p. 5
4.2 ProGlobal: A Global Prediction Model
- ProGlobal predicts each participant’s state at every step using global context and the participant’s previous state, including when the participant is not mentioned.It classifies state as not existing, unknown location, or known location, then predicts a location span for known locations.Source: §4.2 · p. 5
5 Experiments and Analysis
On ProPara, the bAbI-inspired neural baselines are weak on state-change prediction, while ProGlobal outperforms ProLocal, especially on questions requiring step and location predictions. Error analysis identifies implicit state changes, location spans, syntax, and propagation as challenges, motivating several directions for future work.
5.1 Tasks & Evaluation Metrics
- Evaluation asks whether an entity changes, when it changes, and where; answers use accuracy, or F1 for multi-answer questions, with normalized location matching.The three categories cover all participants for whether questions, and only changed participants for when and where questions.Source: §5.1 · p. 7
5.2 Baseline Methods
- Baselines include EntNet and QRN trained on cascaded bAbI-style existence, known-location, and location questions, plus ProComp rules and a logistic-regression/CRF feature model.The cascaded questions are combined at test time to answer the target tasks.Source: §5.2 · p. 7
5.3 Results
- ProGlobal performs substantially better than ProLocal, with gains concentrated in step and location predictions; EntNet and QRN are only slightly above majority baseline on change-type questions.EntNet and QRN are worse than ProLocal on exact-step prediction but better on location prediction.Source: Table 3 · p. 8, §5.3 · p. 8
- The rule-based baseline is limited by incomplete rules and SRL errors, while the feature-based baseline remains poor; 25% of test vocabulary is unseen in train/dev.The authors suggest the feature-based method has not generalized well to unseen vocabulary.Source: §5.3 · p. 8
- A human upper bound of about 80%, based on the best-scoring Turker for each grid, suggests the task is feasible and leaves room for better models.Each grid was annotated by five Turkers.Source: §5.3 · p. 8
5.4 Analysis
- In the example, ProLocal misses destruction after “combine,” while ProGlobal infers sugar’s location from earlier sentences but wrongly places light and CO2 at the leaf from the start.ProLocal ignores surrounding sentences, which can make destructive versus non-destructive uses of “combine” harder to distinguish.Source: §5.4 · pp. 8–9
- Among 100 sampled ProGlobal dev errors, implicit creation or destruction accounted for 37%, location spans 27%, complex syntax 13%, and poor propagation 9%.Location-span errors included wrong spans, overly long spans, and mismatched granularity.Source: §5.4 · p. 9
5.5 Future Directions
- Future directions include global consistency constraints, weakly supervised data augmentation, and backward propagation of state information, which forward-propagating models cannot learn as a bias.The proposed constraints include avoiding creation of existing entities or destruction of nonexistent ones.Source: §5.5 · p. 9
6 Conclusion
The conclusion presents ProPara as a dataset for natural text about a changing world, with entity-state annotations. It also reports challenges for previous models and performance gains from new models using LSTM input encoding and span prediction.
6 Conclusion
- ProPara is the first dataset to pair natural text about a changing world with annotations of entity states during those changes.The authors frame it as a step toward deeper machine understanding in reading comprehension.Source: §6 · p. 9
- Previous models face new challenges on ProPara, while the proposed models exploit surface-level QA ideas to produce performance gains.Source: §6 · p. 9
- The new models use LSTM input encoding and span prediction, and the dataset and models are available to the community.Source: §6 · p. 9