PaperlayerView exact revision on arXiv

Source-linked AI summary

Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos

Olga Loginova, Oleksandr Bezrukov, Frank Keller

arXiv:2610.31117v1cs.CVcs.CL

TL;DR

Procedural mistake recognition asks whether a completed activity deviated from its reference procedure, a reasoning task distinct from temporal segmentation and action recognition. PRISM grounds video segments in language and uses procedural-state decoding; across three benchmarks, its trained local decoder improves over nearest-candidate matching, while SRL improves Any-F1 in all six matched grounded comparisons.

  • Problem

    Post-completion mistake recognition asks whether a completed execution deviates from a reference procedure and where; this problem remains underexplored and distinct from temporal segmentation and action recognition.

    Source: §1 · p. 1

  • Method

    PRISM uses a video-to-language-to-reasoning pipeline, with an HMM mapping description embeddings to task-local Step IDs and reference alignment and semantic checks for mistake recognition.

    Source: §1 · p. 2, §3.4 · p. 5, §5 · p. 8

  • Results

    Across three benchmarks, the trained local decoder consistently improves over nearest-candidate matching, sequence decoding adds dataset-dependent gains, and SRL improves Any-F1 in all six matched grounded comparisons.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The unified edit-based evaluation makes representation and decoding effects comparable across benchmarks.

    Source: §5 · p. 8

    Evaluation assumes benchmark-provided step boundaries, candidate descriptions, and valid references; deployment also requires reliable segmentation and grounding.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Procedural mistake recognition requires identifying deviations from valid multi-step activities. We study post-completion recognition from videos with benchmark-provided step boundaries. We introduce PRISM (Procedural mistake Recognition via Interpretable Semantic Modeling), which grounds video segments in language and maps their descriptions to procedural states using a lightweight hidden Markov model trained on correct executions. Reference alignment and semantic checks identify errors in step content and order. To support evaluation across heterogeneous benchmarks, we map dataset-specific annotations to four edit-based error types: insertion, deletion, substitution, and transposition. Across three benchmarks, we compare gold and VLM-generated descriptions in raw, semantic-role, and action-object forms. Decoder controls show consistent gains from the trained local decoder over nearest-candidate matching, with additional, dataset-dependent benefits from sequence decoding. Semantic role labeling improves Any-F1 in all six matched grounded comparisons, although the gains vary across datasets and do not translate into uniform improvements under separate configuration selection. These results establish language grounding as an effective interface for explicit procedural mistake reasoning and characterize the contributions of representation and decoding.

1 Introduction

Procedural mistake recognition asks whether a completed execution deviates from a reference and where, a problem complicated by fragmented labels and evaluation protocols. PRISM makes step content and order explicit in language, then evaluates shared edit-based error categories and alternative representations and decoding choices.

1 Introduction

  • Post-completion recognition asks whether a completed task deviates from its reference procedure and where, a problem that remains underexplored amid differing error labels, annotation units, and evaluation protocols.The setting is relevant to troubleshooting, quality assurance, training, and human–AI assistance.Source: §1 · p. 1
  • PRISM uses a video-to-language-to-reasoning pipeline in which video segments receive step descriptions that are mapped to task-local Step IDs.The paper compares benchmark-provided gold text with VLM-generated descriptions.Source: §1 · p. 2
  • The framework represents procedural deviations with four edit operations: insertion, deletion, substitution, and transposition.In its representation, Step IDs are HMM states, step descriptions are observations, and state order represents the procedure.Source: §1 · p. 2
  • PRISM combines language grounding, an HMM trained on correct executions, reference alignment, and semantic checks to identify procedural mistakes.Benchmark-provided segment boundaries separate mistake recognition from temporal segmentation.Source: §1 · p. 2
  • The study compares gold and VLM-generated descriptions across raw, semantic-role, and action-object representations, as well as independent and transition-aware inference.These controlled comparisons examine the contributions of representation and decoding.Source: §1 · p. 2

2 Related work

Prior procedural video benchmarks and mistake-recognition systems use heterogeneous error definitions, annotation levels, and description styles. This work targets post-completion recognition in language-grounded step sequences and uses semantic roles as an intermediate normalization layer.

2 Related work

  • Mistake-annotated benchmarks span cooking, assembly, laboratory, and mixed-domain tasks, but vary in error definitions, annotation levels, and description styles.The evaluated benchmarks cover broad task coverage, repeated tasks, and procedures outside cooking.Source: §2 · p. 2
  • Prior mistake-recognition approaches are predominantly vision-centric, while this work performs post-completion reasoning by grounding video executions into textual step sequences.Related work also includes online detection and intervention forecasting.Source: §2 · pp. 2–3
  • Semantic role labeling structures events as predicate–argument representations, and PRISM uses SRLs to normalize free-form step text for mistake reasoning.Prior work has applied semantic structure to video understanding, instruction tracking, and fine-grained mistake attribution.Source: §2 · p. 3

3 Methodology

PRISM models correct procedural steps with HMMs, decodes video-segment representations into step sequences, and compares those sequences with valid references. Language representations, edit alignment, and score-based checks identify structural and atypical-step errors.

3 Methodology

  • PRISM learns what valid steps look like and how they progress from correct executions, then decodes segment representations into task-specific step sequences.It estimates HMM parameters from correct training executions and applies the same task-local candidate inventory and reference-sequence framework across tasks.Source: §3 · p. 3

3.1 HMM training

  • Training projects clip embeddings with PCA and estimates HMM start, transition, and emission parameters from correct executions.Transition and initial-state estimates use Laplace smoothing and local constraints derived from valid reference sequences.Source: §3.1 · p. 3

3.2 HMM inference

  • At inference, Viterbi selects the highest-scoring step sequence using emission and transition terms, with emission scaling adjusting their relative weight.The model computes emission scores for each observation-state pair and records emission and transition scores along the decoded path.Source: §3.2 · p. 4

3.3 Step alignment and mistake recognition

  • Global alignment to the closest valid reference identifies insertions, deletions, transpositions, and substitutions; low emission or transition scores flag atypical steps or transitions.A semantic check can also flag a substitution when an observation differs from its assigned state mean despite sharing its Step ID.Source: §3.3 · p. 5

3.4 Language grounding and step representations

  • PRISM grounds video segments in VLM-generated descriptions or uses benchmark-derived gold text, then represents steps as raw text, semantic roles, or action–object abstractions.SRL preserves arguments such as quantity, destination, surface, and manner; action–object representations discard quantities, attributes, manner, and role relations.Source: §3.4 · pp. 5–6

4 Experiments and results

Experiments evaluate post-completion mistake recognition on three step-segmented benchmarks, comparing representations, decoders, and published systems. The trained local decoder consistently improves over nearest-candidate matching; SRL gains are consistent in matched grounded comparisons but depend on configuration selection.

Benchmarks and protocol

  • The shared protocol uses fully observed, step-segmented videos from CaptainCook4D, EgoPER, and EgoOops, with HMMs trained on correct executions.EgoOops has no official validation split: its HMMs use all 21 correct videos, and configurations transfer from validation results on the other two benchmarks.Source: §4 · p. 6

Metrics

  • Any-F1 is the primary metric for errors labeled insertion, deletion, substitution, or transposition; F1 and AUC are prioritized because errors are imbalanced.Type-F1 requires matching both error location and edit type; AUC excludes gold deletions because they have no observed clip or emission score.Source: §4 · pp. 6–7

Family-level results

  • Gold-text families achieve the highest Any-F1 on every dataset; grounded text variants beat visual embeddings on EgoPER and CaptainCook4D, while vision leads on EgoOops.Among separately selected grounded configurations, Raw has higher Any-F1 than SRL in all six pairs, although SRL improves Seq in four.Source: Table 1 · p. 7, §4 · p. 7

Matched representation comparisons

  • GPT-SRL improves Any-F1 by 0.001–0.036 in all six matched grounded comparisons, with the largest gain for Qwen on EgoOops.Gains on CaptainCook4D are small for both VLMs; the benefit depends on downstream configuration and selection criterion.Source: Table 2 · p. 7, §4 · p. 7

Representation comparisons

  • Act.+Obj. exceeds independently selected configurations in seven of nine dataset–source settings; both exceptions use grounded EgoOops descriptions.Act.+Obj. reports the best configuration within that representation family.Source: §4 · pp. 7–8, Table 2 · p. 7

Decoder controls

  • Greedy emission decoding improves Any-F1 over nearest-candidate matching in all twelve conditions; Viterbi improves seven, matches four, and reduces one.Viterbi exceeds nearest-candidate matching in eleven conditions overall; Any-F1 is unchanged by transition-aware decoding on EgoPER, and its only decrease relative to greedy decoding is Qwen Raw on EgoOops.Source: Table 3 · p. 8, §4 · p. 8

Comparison with prior work

  • On CaptainCook4D, PRISM SRL configurations obtain EDA of 0.841–0.843 and AUC of 0.900; on EgoPER, AEM with ground-truth segmentation exceeds PRISM on both metrics.Segmentation and configuration-selection differences matter when interpreting this system-level comparison.Source: Table 4 · p. 8, §4 · p. 8

5 Conclusion

PRISM combines language grounding, an HMM trained on correct executions, reference alignment, and semantic checks for post-completion procedural mistake recognition. Across three benchmarks, trained local decoding consistently beats nearest-candidate matching, while sequence decoding and SRL benefits vary by dataset and configuration.

Conclusion

  • PRISM combines an HMM trained on correct executions with reference alignment and semantic checks to recognize procedural mistakes after task completion.Across three benchmarks, trained local decoding consistently improves over nearest-candidate matching, while sequence decoding provides dataset-dependent additional gains.Source: §5 · p. 8
  • SRL improves Any-F1 in all six matched grounded comparisons, but its benefits depend on dataset and downstream configuration.The unified edit-based evaluation makes these effects comparable across benchmarks.Source: §5 · p. 8
  • The results support language grounding as an effective interface for reasoning explicitly about procedural step content and order.The framework makes the underlying deviation explicit, including when a failure appears only as overboiled zoodles.Source: §5 · p. 9

Limitations

The evaluation depends on benchmark-provided boundaries, task-specific candidate descriptions, and valid reference procedures. The edit taxonomy also trades semantic detail for cross-dataset compatibility, while SRL reasoning assumes consistent role normalization.

Limitations

  • Evaluation assumes benchmark-provided step boundaries, task-specific candidate descriptions, and valid reference procedures.Grounded experiments also depend on the VLM preserving error-relevant details.Source: Limitations · p. 9
  • SRL-based reasoning assumes consistent role normalization, since inconsistent semantic parses can propagate into HMM reasoning.Source: Limitations · p. 9
  • The four edit labels merge semantically distinct cases, including Correction with Insertion and Wrong Execution with Substitution.The authors propose refining substitutions into full-step substitutions and role-level wrong executions.Source: Limitations · p. 9

Ethical considerations

The work uses existing benchmarks without human-subject experiments or real-world deployment, and flags risks from video-based monitoring and downstream grounding errors. Any deployment calls for human oversight and privacy safeguards.

Ethical considerations

  • The work uses existing benchmark datasets and includes no new data collection, human-subject experiments, or real-world decision-making deployment.Source: Ethical considerations · p. 9
  • Because generated step-description errors may propagate into misleading labels, the authors recommend human oversight, privacy, consent, and secure data handling in deployment.They advise against using the system alone for sanctions or safety-critical decisions.Source: Ethical considerations · p. 9

A Dataset-specific annotation profiles

The three benchmarks differ in scale, domain, and annotation style, shaping how step descriptions and errors are represented. The section profiles those differences and maps native annotations to a shared edit taxonomy for evaluation.

A Dataset-specific annotation profiles

  • The benchmark profiles compare video duration and annotated step instances across datasets whose papers use different names for these units.Source: Table 5 · p. 11, A Dataset-specific annotation profiles · p. 11
  • CaptainCook4D has noisy, near-duplicate step descriptions, while argument changes in quantity, duration, instrument, or destination can distinguish modified steps.SRL preserves these argument-level distinctions explicitly.Source: A Dataset-specific annotation profiles · pp. 11–12
  • EgoPER uses short, regular instructions and often local action, object, or destination substitutions; action–object abstractions lose useful distinctions.Text and SRL remain close in performance, while the abstraction drop is less pronounced than in CaptainCook4D or EgoOops.Source: A Dataset-specific annotation profiles · p. 12
  • EgoOops combines heterogeneous, sometimes multi-event steps with short error cues, making role fillers and argument structure especially salient.The authors append cues to canonical steps to construct modified descriptions.Source: A Dataset-specific annotation profiles · p. 12
  • Gemini usually grounds descriptions more accurately than Qwen on CaptainCook4D, while EgoOops remains difficult because of long compositional steps and added error events.On EgoPER, both VLMs are closer to the candidate-step vocabulary.Source: A Dataset-specific annotation profiles · p. 12

B Unified label mapping

  • Dataset-specific annotations are mapped to Insertion, Deletion, Substitution, and Transposition as gold targets for Type-F1, without changing HMM training or Step-ID decoding.Source: B Unified label mapping · p. 12
  • When several valid references exist, evaluation selects the reference closest to the decoded sequence before instantiating structural targets.Source: B Unified label mapping · p. 12
  • The mapping handles native conventions differently: EgoPER transpositions are derived by sequence alignment, while EgoOops lacks native Deletion and Transposition labels.EgoOops's rare “others” cases are resolved individually from their textual descriptions.Source: B Unified label mapping · pp. 12–13

C Semantic representation construction

The paper constructs semantic-role representations to normalize step descriptions into explicit predicate–argument structures. Two generators share a constrained format but differ in their role inventories and encoding choices, while prompt calibration uses split-independent descriptions.

C Semantic representation construction

  • Each step's SRL representation makes its main event, participants, and key attributes explicit, using nested structures when meaning requires internal attachment, scope, or composition.Source: C Semantic representation construction · p. 18
  • GPT-5.3 and Qwen3.6-27B generate SRL offline with the same constrained few-shot prompt and output fields, but use different decoding and output-enforcement settings.Source: C Semantic representation construction · p. 18
  • GPT-5.3 tends to preserve surface predicates, whereas Qwen3.6-27B more often rewrites instructions into normalized action representations.Source: C Semantic representation construction · p. 18
  • GPT-5.3 uses 98 distinct role labels and Qwen3.6-27B uses 71, with differences in how spatial, temporal, and quantity distinctions are represented.Qwen3.6-27B more often encodes distinctions with general roles or nested modifiers.Source: C Semantic representation construction · p. 18
  • Prompt calibration uses ontology descriptions without split membership, mistake labels, HMM predictions, or evaluation results; each unique description is converted once across datasets.Manual inspection led to schema clarifications or additional examples, producing a final nine-example prompt.Source: C Semantic representation construction · p. 18

D HMM hyperparameters and computational details

The appendix documents experiment configurations and computational costs, then examines grounding, representation, error-type, uncertainty, and multimodal results. SRL gains and multimodal trade-offs vary by dataset and condition.

D HMM hyperparameters and computational details

  • The largest benchmark’s multimodal sweeps consumed approximately 57 node-hours, while all experiments together consumed several hundred node-hours.HMM training and inference are CPU-bound and take seconds per task; video-to-text grounding with Qwen 3.5-27B on the largest benchmark took roughly 24 GPU-hours.Source: D HMM hyperparameters and… · p. 18

E.1 Family-level configurations

  • Family-level configurations use validation Any-F1 on CaptainCook4D and EgoPER; EgoOops transfers the best mean validation Any-F1 configuration across those datasets, without using EgoOops error-video labels.All test metrics in a leaderboard row come from the same configuration.Source: §E.1 · p. 19

E.3 Prior-work comparison configurations

  • PRISM configurations for the system comparison use validation EDA on CaptainCook4D and EgoPER; EgoPER validation measures correct-segment acceptance, not error recall.Test metrics are not used for selection, and each row’s benchmark-specific metrics share one configuration.Source: §E.3 · p. 19

F.1 Grounding accuracy

  • The grounding appendix distinguishes word-for-word Exact from Seq, which measures Step-ID recovery after alignment; non-verbatim descriptions can still yield the correct Step ID.A separate positionwise Step-ID accuracy analysis is reported without sequence alignment.Source: §F.1 · pp. 19–20

F.2 Second SRL generator

  • Qwen-SRL improves five of six matched Raw conditions; GPT-SRL gives higher Any-F1 in five, with a rounded tie on CaptainCook4D using Gemini grounding.Gains occur with both SRL generators, but their magnitude varies across conditions.Source: §F.2 · p. 20

F.3 Stratified SRL analysis; F.4 Performance by error type

  • The appendix defines positionwise strata for decoded-ID changes and reports type-specific F1 and support; EgoOops has no gold Deletion or Transposition instances, whose F1 is zero by convention.Structural edits use the selected valid reference; a transposition pairs an insertion and deletion of the same Step ID at different alignment positions.Source: §F.3 · p. 20, §F.4 · p. 20

F.5 Bootstrap confidence intervals

  • SRL-gain intervals exclude zero for Qwen on EgoOops and both VLMs on EgoPER, but include zero for Gemini on EgoOops and both CaptainCook4D comparisons.Point estimates are positive in all six comparisons; the intervals do not establish superiority over published systems whose per-video predictions are unavailable.Source: §F.5 · p. 21

F.6 Multimodal fusion sweep

  • Text-heavy mixtures are strongest on EgoOops; added visual weight slightly raises Any-F1 but lowers Seq and AUC on CaptainCook4D, while raising Seq and AUC with nearly flat Any-F1 on EgoPER.The authors do not present multimodal fusion as a universal solution; in this HMM setting, textual embeddings anchor the state space more sharply than frozen visual features.Source: §F.6 · p. 21

F.7 Multimodal results with Qwen3.6-27B SRL

Matched multimodal runs show no consistent average advantage for either SRL generator. The qualitative appendix illustrates what SRL and action–object representations preserve, and how a shared edit taxonomy applies across datasets.

F.7 Multimodal results with Qwen3.6-27B SRL

  • Across matched multimodal runs, GPT-SRL and Qwen-SRL have mean differences close to zero on all three datasets, with neither showing a consistent average advantage.Individual configurations may respond differently.Source: §F.7 · p. 21

G.1 Error-relevant semantic distinctions

  • SRL can explicitly represent error-relevant distinctions including relations, destinations, materials, and nested events.The qualitative appendix examines when SRL helps over grounded text without SRL, what action–object abstraction preserves or loses, and how the unified edit-based view applies across datasets.Source: G Additional qualitative examples · p. 21, §G.1 · p. 22

G.2 What action–object preserves and loses

  • Action–object representations retain the main action and all visually involved objects, including tools and contacted objects, but can lose duration, quantity, manner, and multi-action detail.This makes the abstraction informative for some mistakes but too coarse for others; the appendix highlights manner-sensitive steps in EgoPER.Source: §G.2 · p. 22
  • In EgoPER, short, regular candidate descriptions limit lexical variation, so Raw and SRL remain close; richer text may contrast more with compressed action–object forms.The example contrasts a manner-sensitive instruction with its action–object representation.Source: §G.2 · p. 22

G.3 Examples of the unified error taxonomy

  • The four-type edit taxonomy remains interpretable across datasets despite differences in dataset-specific error labels and annotation styles.Source: §G.3 · p. 22

H Decoder Ablations

Decoder controls compare independent nearest-candidate matching, greedy decoding with trained local emissions, and Viterbi decoding with sequence information. Greedy decoding consistently improves Any-F1 over nearest-candidate matching, while Viterbi’s effects vary by dataset and condition.

H Decoder Ablations

  • The nearest-candidate baseline assigns each segment independently by cosine similarity, while the greedy decoder retains matched PRISM settings and changes only the decoding rule.The nearest-candidate decoder uses native, L2-normalized encoder embeddings; the greedy decoder uses the fixed downstream configuration.Source: H Decoder Ablations · pp. 23–24
  • Nearest-candidate matching uses frozen language representations, task-specific candidates, and edit alignment without a trained HMM; its comparison with greedy decoding adds PCA and a trained local emission model.Greedy-to-Viterbi changes the decoding rule to add the sequence prior; Viterbi adds start and transition probabilities and global sequence search.Source: H Decoder Ablations · p. 23, Table 27 · p. 27
  • Greedy decoding improves Any-F1 over nearest-candidate matching in all twelve conditions; Viterbi improves it in some conditions, leaves all four EgoPER conditions unchanged, and lowers Qwen Raw on EgoOops from 0.339 to 0.321.On CaptainCook4D, Viterbi improves Any-F1 while lowering Seq.Source: H Decoder Ablations · p. 24, Table 27 · p. 27

Related papers

Loading 2610.31117v1…