Source-linked AI summary
Interscript: A dataset for interactive learning of scripts through error feedback
Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, Yiming Yang
TL;DR
Structured prediction models can generate inconsistent outputs, while prior feedback-learning evidence is concentrated in synthetic or constrained settings. The paper introduces INTERSCRIPT, a crowdsourced dataset of 8,466 erroneous-script, feedback, and corrected-script tuples for interactive learning. Its motivating study finds that scripts that look good can still contain commonsense errors, and the dataset is positioned as a first step toward interactive correction of deployed structured models.
Problem
Real-world datasets for feedback-driven correction of structured prediction errors are lacking, despite models producing semantically inconsistent outputs.
Method
The paper introduces INTERSCRIPT, a crowdsourced dataset of 8,466 tuples linking an erroneous script, user feedback, and a corrected script.
Results
The initial study found that scripts appearing high-quality often contained commonsense mistakes, revealing a gap between syntactic and semantic correctness.
Takeaways & Limitations
INTERSCRIPT provides a first real-world resource for interactive script generation through error feedback and outlines use-cases for graph correction and continuous learning.
Takeaways & Limitations
The dataset collection assumes that average users can identify mistakes because many generated-script errors reflect missing basic commonsense.
Abstract
from arXiv · showhide
How can an end-user provide feedback if a deployed structured prediction model generates inconsistent output, ignoring the structural complexity of human language? This is an emerging topic with recent progress in synthetic or constrained settings, and the next big leap would require testing and tuning models in real-world settings. We present a new dataset, Interscript, containing user feedback on a deployed model that generates complex everyday tasks. Interscript contains 8,466 data points -- the input is a possibly erroneous script and a user feedback, and the output is a modified script. We posit two use-cases of \ours that might significantly advance the state-of-the-art in interactive learning. The dataset is available at: https://github.com/allenai/interscript.
1 Introduction
Structured prediction models can produce outputs that are formally plausible but semantically inconsistent, motivating real-world feedback datasets. INTERSCRIPT addresses this gap with crowdsourced error-feedback examples for interactive correction.
- At least 20% of generated scripts contain commonsense mistakes that average users can identify, despite appearing ostensibly valid.The motivating error involves an implausible ordering of driving and getting in a car.
- Existing feedback-driven correction datasets are limited by synthetic or controlled tasks and therefore do not provide broadly applicable real-world evidence.
- INTERSCRIPT contains 8,466 crowdsourced examples pairing an erroneous script and feedback with a corrected script.The dataset includes explicit, implicit, and distractor feedback types.
- The dataset is intended to support interactive graph correction and continuous learning for structured prediction.The authors describe these as two use-cases that might advance interactive learning.
- The work frames INTERSCRIPT as a first step toward interactive machine-learning systems for widespread structured prediction tasks.
2 Related work
Prior interactive-learning work uses human feedback, but structured-task datasets often constrain feedback or make it task-specific. INTERSCRIPT instead targets natural, non-actionable, and reusable feedback for complex structures.
- Language is an expressive interaction mode for correcting models, but controlled feedback settings do not generalize well to the richness of real-world human feedback.
- Prior structured correction systems map feedback into structure-edit commands, often requiring task-specific formal feedback that directly specifies the fix.NLEDIT, for example, can struggle with natural feedback that only highlights a problem.
- INTERSCRIPT makes no assumptions about feedback structure and treats feedback as non-actionable, allowing users to identify errors without enumerating repairs.
- Unlike query-specific feedback in NLEDIT, INTERSCRIPT feedback addresses general commonsense errors and can apply to other scripts with similar issues.
3 Inspiration: Script generation
Script generation asks models to construct partially ordered event graphs for everyday goals, but generated graphs can contain commonsense and ordering errors. The initial study motivates feedback-based correction by showing that these errors are recognizable and often locally repairable.
- 3.1 Script generation: Script generation models jointly generate essential events and temporal ordering constraints as a partially ordered graph G(V, E).An edge e_ij means event v_i must precede event v_j, while some events may occur in either order.
- 3.1 Script generation: PROSCRIPTgen uses an 11B-parameter T5-XXL model to generate scripts in DOT graph format, with generally high-quality events and structure.
- 3.1 Script generation: Mechanical Turk workers corrected most generated scripts within a few edits, making interactive learning attractive because outputs expose the model’s beliefs about a goal.Typical correction distance was 5 edits.
- 3.3 Conclusions from the initial study: The sampled graphs averaged about two mistakes, often using an entity before it was obtained, suggesting that similar feedback may transfer across examples.
- 3.3 Conclusions from the initial study: The scripts looked good on the surface, but closer inspection found that most errors reflected missing basic commonsense, exposing a syntax–semantics gap.
- 3.3 Conclusions from the initial study: The study asks whether general commonsense principles explain model failures and how models can incorporate simple feedback without costly retraining.It identifies the absence of a real-world structured-prediction dataset for this interactive-learning problem.
4 INTERSCRIPT collection
INTERSCRIPT was collected through crowdsourced annotation of model-generated and gold ProScript scripts, focusing annotators on serious commonsense errors and structured feedback. The resulting dataset combines multiple feedback forms, distractors, and shared-feedback examples into 8,466 samples.
- Data selection: The collection began from ProScript test examples with predicted graph edit distance ≥8, yielding about 400 predicted-graph and gold-script annotation inputs.Cases in which annotators marked the gold script as worse were later removed.
- Annotation task: Annotators compared generated and gold scripts, identified one serious commonsense error, and recorded its type, location, explanation, and general principle.Feedback included missing steps, wrong steps, wrong order, wrong partial order, localized node or edge identifiers, and explanations intended for a five-year-old.
- Annotation quality: Each data point received three annotations, and agreement reached Fleiss Kappa 0.90 for graph labels and 0.75 for explicit feedback.The authors also report diversity in which serious mistake annotators selected.
- Distractor feedback: Distractor feedback was created by attaching feedback from lexically related but least-similar graph examples, modeling irrelevant user feedback.Graph similarity was estimated with Sentence Transformers embeddings and cosine distance.
- Shared feedback: Shared-feedback examples were synthesized by substituting related goal words and manually labeling 50 resulting examples.This construction tests whether feedback about one situation can transfer to a similar situation.
5 Potential Use Cases
INTERSCRIPT is intended to support interactive systems that respond to user feedback after deployment and reuse feedback across related errors. The paper highlights learning to apply or reject feedback and maintaining a memory of prior failures as two potential use cases.
- Learning to apply feedback: One use case trains a model to apply user feedback or determine that it is not applicable in a real-world structured prediction setting.This could let deployed systems correct answers after deployment.
- Maintaining feedback memory: A second use case maintains memories of errors and corrections so feedback from one context can inform responses in a different but similar context.The paper connects this idea to recurring situations and recursive reminding in psychology.
- Maintaining feedback memory: INTERSCRIPT feedback often expresses general principles, such as getting into a vehicle before driving, that can transfer across related situations.The paper illustrates analogous errors involving closed containers and missing opening steps.
6 Conclusion
INTERSCRIPT is presented as the first real-world dataset for interactive script generation through error feedback, containing 8,466 data points and supporting two potential use cases for interactive machine learning.
- INTERSCRIPT contains 8,466 data points with a rich hierarchy of feedback types and is intended to enable two use cases for interactive machine learning.The authors emphasize its potential relevance to very large language models that are difficult to retrain after deployment.