Source-linked AI summary
Simulating Action Dynamics with Neural Process Networks
Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, Yejin Choi
TL;DR
Procedural language understanding requires anticipating action effects on entities that are not stated explicitly. Neural Process Networks simulate actions on entity states using learned action operators and recurrent memory. In the cooking domain, the model learns action semantics and tracks dynamic state changes, with advantages over competitive baselines.
Problem
Procedural text requires anticipating implicit causal effects of actions on entities, which most natural-language understanding algorithms cannot reason about.
Source: §1 · p. 1Method
Neural Process Networks apply learned action operators to entity representations and use recurrent memory to track state changes as they read text.
Source: §1 · p. 1, §2.1 · p. 2Results
In the cooking domain, the model learns action semantics and tracks dynamic entity state changes, showing advantages over competitive baselines.
Source: §7 · p. 10Takeaways & Limitations
The model can reason about unstated action effects and provide more informative contexts for procedural-text understanding and generation, with more interpretable internal representations.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated. In this work, we introduce Neural Process Networks to understand procedural text through (neural) simulation of action dynamics. Our model complements existing memory architectures with dynamic entity tracking by explicitly modeling actions as state transformers. The model updates the states of the entities by executing learned action operators. Empirical results demonstrate that our proposed model can reason about the unstated causal effects of actions, allowing it to provide more accurate contextual information for understanding and generating procedural text, all while offering more interpretable internal representations than existing alternatives.
1 INTRODUCTION
Procedural text understanding requires anticipating implicit action effects on entities, a capability most natural-language systems lack. Neural Process Networks simulate actions on entity representations, learning action operators from text and reporting more informative context representations, interpretability, and a new annotated dataset.
1 INTRODUCTION
- Most natural-language understanding algorithms cannot reason about causal effects that procedural text leaves unstated.Understanding instructions or stories requires anticipating those implicit effects on entities.Source: §1 · p. 1
- Neural Process Networks read sentences sequentially, select actions and affected entities, and retain induced state changes in recurrent memory.For example, the model applies learned “wash” and “cut” functions to a tomato embedding to reason about shape and cleanliness.Source: §1 · p. 1
- The model learns explicit action representations as functional operators from text, predicting entity attributes to explain action effects.This textual simulation addresses state dimensions such as cleanliness and composition that spatially focused virtual environments may omit.Source: §1 · p. 2
- Explicit action embeddings provide an inductive bias for more informative context representations in procedural-text understanding and generation.The paper also presents a dataset with fine-grained state-change annotations for future research.Source: §1 · p. 2
2 NEURAL PROCESS NETWORK
The Neural Process Network reads procedural sentences sequentially and simulates actions on relevant entities through learned representations. Its modules select actions and entities, update entity states, and predict resulting state changes.
2.1 OVERVIEW AND NOTATION
- Entity-state embeddings are initialized for document entities, then selected actions transform selected entities; prediction errors on end states train action representations to model causal effects.The model reads one sentence at a time, and updated embeddings encode entity state attributes.Source: §2.1 · p. 2
2.2 SENTENCE ENCODER
- A GRU encodes each word and uses its last hidden vector as the sentence representation.Source: §2.2 · p. 2
2.3 ACTION SELECTOR
- The action selector uses the sentence encoding to choose actions, composing multiple selected action embeddings as a weighted average.For example, “wash and cut” can select both actions.Source: §2.3 · pp. 2–3
2.4 ENTITY SELECTOR
- The entity selector attends to relevant entities and can carry prior sentence attention forward to handle elided arguments or pronouns.Its final attention combines current attention, previous attention, and an option to select no entity.Source: §2.4 · pp. 3–4
2.5 SIMULATION MODULE
- The simulation module applies selected actions to selected entity representations, then interpolates the proposed representation with current entity embeddings according to attention.Attention can allow an entity’s state embedding to be completely overwritten.Source: §2.5 · p. 4
2.6 STATE PREDICTORS
- State predictors classify the proposed entity representation along six dimensions; location prediction also uses the original sentence representation.The dimensions are location, cookedness, temperature, composition, shape, and cleanliness.Source: §2.6 · p. 4
3 TRAINING
Training uses an action lexicon with annotated state changes, recipe data, and component-wise objectives. Because dense annotations are expensive at scale, the model also uses heuristically derived weak supervision.
3.1 STATE CHANGE KNOWLEDGE
- The action lexicon covers 384 cooking actions and six state-change dimensions, with crowdsourced annotations of induced changes and end states.Possible end states range from two for binary changes to more than 200; for example, bake affects temperature, cookedness, and location.Source: §3.1 · p. 4, Table 1 · p. 5
3.2 DATASET
- Training uses 65,816 recipes, with 175 for development and 700 for testing; development and test sentences have dense crowdsourced annotations of actions, entities, and state changes.Source: §3.2 · p. 5
3.3 COMPONENT-WISE TRAINING
- Component-wise objectives train action selection, entity selection, and state prediction, while weak supervision derives labels from recipe verbs and their induced state changes.A coverage loss penalizes narratives whose combined attention for an entity does not sum to more than 1; the attention weights are bounded by 1.Source: §3.3 · p. 5
4 EXPERIMENTAL SETUP
The experiments assess entity and state-change tracking in recipes, then test whether encoded entity states support next-step generation. The section specifies task metrics, baselines, ablations, and generation setup.
4 EXPERIMENTAL SETUP
- The evaluation combines intrinsic entity and state-change tracking with an extrinsic test of generating future recipe steps from prior steps.The experiments also include qualitative analysis of the model’s internal components.Source: §4 · p. 5
4.1 INTRINSIC EVALUATION - TRACKING
- Tracking evaluates entity selection and end-state prediction, including entities that have been combined or remain uncombined.Entity-selection F1 uses attention above 50% to identify selected entities; state-change scores average F1 and accuracy across six types.Source: §4.1 · pp. 5–6
- Tracking compares the model with an independently trained GRU and an adapted Recurrent Entity Network, and examines six architectural and training ablations.Ablations remove recurrent attention, coverage loss, action-selector-to-entity-selector connections, or action-selector pretraining, and test skip-gram-initialized action embeddings.Source: §4.1 · p. 6
4.2 EXTRINSIC EVALUATION - GENERATION
- Generation tests whether the model can produce the next recipe step using all previously performed steps as context.The model encodes its simulated entity-state vectors and context words with separate bidirectional GRUs before initializing the decoder.Source: §4.2 · pp. 6–7
- Generation is evaluated with corpus-level BLEU and ROUGE, plus VF1 for action overlap and SF1 for end-state overlap against the reference.Baselines are non-attentive and attentive seq2seq models and a generator using entity states from the Recurrent Entity Network.Source: §4.2 · pp. 6–7
5 EXPERIMENTAL RESULTS
The intrinsic evaluations report strong entity selection, comparative state tracking, action-embedding patterns, and entity-composition behavior. In generation, adding NPN entity states improves scores over competitive baselines and supports predictions conditioned on ingredient states.
5.1 INTRINSIC EVALUATIONS
- 55.39% F1: the full model outperforms all baselines on entity selection.It handles uncomposed and composed entities, though it can fail to map references such as “pizza” or “dough” to their constituent entities.Source: §5.1 · p. 7
- The full model outperforms competitive baselines on state-change tracking; the coverage-loss ablation has higher accuracy while predicting fewer changes.Initializing action embeddings with skipgram vectors and locking them also shows relatively high performance.Source: §5.1 · p. 8
- Actions with similar functions are neighbors in embedding space, suggesting the learned representations capture semantic properties through their state transformations.The paper reports examples such as “cut” and “slice.”Source: §5.1 · p. 8, Table 4 · p. 8
- Entity-state cosine similarity increases more for entities combined in a sentence (blue) than for those not combined (red), despite similarity leakage from soft attention.The authors conclude that the model generally captures compositionality patterns from entity combinations.Source: §5.1 · p. 8, Figure 3 · p. 8
5.2 EXTRINSIC EVALUATIONS
- Sequences generated with NPN entity states score higher than competitive baselines, and higher VF1 and SF1 indicate better next-action prediction.Table 6 examples show conditioning on ingredient states, such as raw butter likely being melted or cooked flan refrigerated.Source: §5.2 · pp. 8–9, Table 5 · p. 8, Table 6 · p. 9
- Seq2seq and EntNet generations follow immediate context, while the NPN-aware examples reflect broader ingredient-state patterns.The examples include draining cooked squash even when draining is irrelevant to the recipe step.Source: §5.2 · p. 9
6 RELATED WORK
Related work spans neural memory and entity-state tracking, function-based reasoning, recipe graphs, and visual or multimodal state tracking. This work instead learns action representations from text and models their effects on entity states.
6 RELATED WORK
- Neural memory and machine-reading systems store running text representations, while entity-tracking work adds state tracking; this paper represents action effects on entities directly.The cited work includes memory architectures [15, 16, 17, 18] and Recurrent Entity Networks [19].Source: §6 · p. 9
- Neural Programmers select database rows and apply predefined operations [20, 21], whereas this framework learns representations for a broader set of actions.The learned action representations model how actions change the state space.Source: §6 · p. 9
- Recipe-graph studies build discrete representations with probabilistic models [22, 23, 24], while this model integrates action and entity relations and tracks entity-state changes.Source: §6 · p. 9
- Prior visual or multimodal work models state changes through graphs or agent decisions in videos and games [25, 26, 27, 28, 29, 30], while this work uses text-based signals and embedding-space transformations.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents Neural Process Networks as learning action transformations that update entity-state representations and track dynamic changes. In the cooking domain, the model learns action effects and shows advantages over competitive baselines.
7 CONCLUSION
- Neural Process Networks learn action transformations to model causal effects on entity states, maintain recurrent state memory, and predict state changes; cooking-domain results show advantages over competitive baselines.Source: §7 · p. 10
A TRAINING DETAILS OF OUR FULL MODEL AND ABLATIONS
The appendix reports separate training configurations for tracking models and the generation model. Tracking uses distinct entity and action selectors, while generation jointly projects context and state representations.
A.1 TRACKING MODELS
- Tracking models use a 100-unit instruction encoder, 30-dimensional action and entity embeddings, dropout 0.3, and Adam at 0.001; separate instruction encoders serve entity and action selection.Word and entity embeddings are initialized with skipgram vectors; coverage-loss gradients apply only when no entity is annotated, and entity-selection-loss gradients are zeroed when no entity label is present.Source: §A.1 · p. 13
A.2 GENERATION MODEL
- The generation model uses 200-unit context and state encoders with 30-dimensional state vectors, dropout 0.3, and an initial learning rate of 0.0003 halved every five epochs.Context and state representations are combined by element-wise product followed by a linear projection.Source: §A.2 · p. 13
B TRAINING DETAILS OF BASELINES
This section documents tracking and generation baseline configurations, then describes how state-change labels, end states, and development/test annotations were collected.
B.1 TRACKING BASELINES
- Tracking baselines include a GRU with separate entity-selection and state-change-prediction encoders, and an EntNet with memory cells tied to document entities.The EntNet initializes one entity cell per ingredient; the GRU uses 100-dimensional hidden states.Source: §B.1 · p. 13
B.2 GENERATION BASELINES
- Generation baselines include a bidirectional GRU seq2seq model, attentive seq2seq with context attention at each decoder step, and an EntNet generator using EntNet-produced state representations.The seq2seq model minimizes next-word negative log-likelihood.Source: §B.2 · pp. 13–14
C.1 ANNOTATING STATE CHANGES
- Workers assigned state-change types to verbs by majority vote; 342 of 384 verbs received a type, and 74 of those had multiple types.Seven workers annotated each verb.Source: §C.1 · p. 14
C.2 ANNOTATING END STATES
- End-state labels were manually aggregated by state-change type from worker-provided examples, including locations such as “pan,” “pot,” or “oven.”Table 7 lists end states for each state-change type.Source: §C.2 · p. 14, Table 7 · p. 14
C.3 ANNOTATING DEVELOPMENT AND TEST SETS
- Development and test annotations mark entities undergoing one of six state changes, newly created ingredient combinations, and actions included by majority vote.“Cut the tomatoes and add to the onions” illustrates a SHAPE change and a tomatoes-onions combination.Source: §C.3 · p. 14
D.1 REMOVING TRAINING DATA
Table 8 reports development-set entity-selection and state-change-selection results when training labels are randomly dropped at varying percentages.
D.1 REMOVING TRAINING DATA
- Table 8 covers development-set entity selection and state-change selection under random dropping of training labels by percentage.The caption gives the table’s scope but no result values.Source: Table 8 · p. 15