Source-linked AI summary
Post-Training VLMs for Video Mistake Detection
Federico Spurio, Olga Zatsarynna, Lars Doorenbos, Emad Bahrami, Gianpiero Francesca, Juergen Gall
TL;DR
The paper addresses whether video-language models can detect instructional mistakes beyond closed-set, step-specific settings. It introduces the MD-VQA benchmark and a reward-driven VLM post-training method, which achieves state-of-the-art performance, particularly on unseen scenarios. The benchmark remains challenging because absolute performance leaves substantial room for improvement.
Problem
Closed-set mistake-detection methods do not test whether models learn a general concept of mistakes across changed or unseen instructional steps.
Method
The paper introduces MD-VQA and a VLM post-training method with a tailored reward that encourages identifying mismatches between instructions and videos.
Results
The post-training method achieves state-of-the-art performance against zero-shot, fine-tuning, and post-training baselines, especially on steps never observed during training.
Takeaways & Limitations
MD-VQA provides a protocol for assessing mistake-detection generalization across seen and unseen instructional steps.
Takeaways & Limitations
Absolute benchmark performance remains limited, and the task still requires considerable improvement.
Abstract
from arXiv · showhide
Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.
1 Introduction
The paper introduces MD-VQA to evaluate whether video-language models can detect instructional mistakes in both seen and unseen steps. It proposes a first VLM post-training technique whose tailored reward improves mistake detection, especially in unseen scenarios.
- The benchmark addresses the limitation of closed-set protocols, where changes to tasks require new data collection and model retraining.
- The benchmark combines CaptainCook4D and EgoPER into diverse questions spanning complex instructional steps and mistake types.
- 5.0 F1-score points, or 11.6%, separate the proposed post-training method from the next-best baseline, especially on unseen scenarios.
- MD-VQA evaluates whether models can identify mistakes in both familiar and unseen instructional steps.
- The proposed method is the first VLM post-training technique designed to address MD-VQA.
2 Related Work
Prior work studies video mistakes through online detection, temporal segmentation, classification, and zero-shot long-form understanding. Reinforcement-learning post-training methods, including GRPO, provide an efficient alternative to critic-based PPO for VLM alignment.
- Prior mistake-understanding work includes online error detection, temporal action and error segmentation, per-clip classification, and zero-shot long-form detection.
- PPO-based reinforcement learning can improve VLM alignment but relies on an additional critic model.
- GRPO removes the critic network and computes advantages relative to group mean rewards, offering a computationally efficient alternative.
3 MD-VQA: Video Question Answering for Mistake Detection
MD-VQA formulates stepwise mistake detection as binary video question answering over an instruction and its corresponding clip. The benchmark repurposes two datasets, evaluates seen and unseen activities, and measures recall, precision, and F1-score.
- 3.1 MD-VQA Task: MD-VQA asks whether a video execution matches a textual instructional step, using binary yes/no answers for mismatch detection.
- 3.1 MD-VQA Task: Models must recognize mistakes in familiar and unseen instructional steps rather than rely on a predefined step set.
- 3.2 Data: The benchmark repurposes CaptainCook4D and EgoPER, covering fine-grained recipe steps with correct and erroneous executions.
- 3.2 Data: EP-VQA defines Slip, Technique, Measurement, and Utensil as four high-level mistake categories.
- 3.3 Evaluation: Unseen test splits contain clips from activities excluded from training, while seen splits use the remaining data for evaluation.
- 3.3 Evaluation: Performance is evaluated with recall, precision, and F1-score, treating mistakes as the positive class.
4 Method
The method post-trains VLMs with GRPO and rewards that combine response format, answer accuracy, and sensitivity to opposite correct or incorrect executions. It uses paired clips and an accuracy-gated opposite reward to encourage detecting discrepancies between instructional descriptions and videos.
- 4.1 Preliminaries: GRPO samples groups of responses, assigns rule-based rewards, and computes per-response advantages relative to other candidates.The policy is optimized with a clipped objective and a KL regularizer relative to a fixed reference policy initialized from the instruction-tuned model.
- Method overview: The method post-trains VLMs with GRPO using format, accuracy, and a novel opposite reward for MD-VQA.The format reward supports structured reasoning, the accuracy reward rewards correct answers, and the opposite reward promotes different answers for paired clips showing contrasting executions.
- Format reward: The format reward requires the predefined output structure, while the thinking format encloses reasoning traces in <think></think> and answers in <answer></answer>.This format is intended to encourage intermediate reasoning over textual and visual tokens before the final answer.
- Opposite reward: The opposite reward compares each clip with a sampled clip showing the opposite execution of the same instructional step.A correct clip is paired with a failed execution and vice versa; the reward is constrained by answer accuracy so contrasting responses do not come from simply reversing labels.
- Final reward: The final reward combines format and accuracy rewards with the opposite reward, whose influence is controlled by λ.The opposite reward is granted when the paired answers differ and the answer for the original clip matches the ground truth.
5 Experiments
Experiments compare zero-shot, supervised fine-tuning, and post-training methods across seen and unseen splits, with ablations and qualitative examples. The proposed method generally improves mistake detection and transfer to unseen instructions, while broader procedural context remains a limitation.
- Baselines: The benchmark compares zero-shot, reasoning-prompt, supervised fine-tuning, explanation-augmented, GRPO, and the proposed post-training baselines.All evaluated methods use the same base architecture, with additional GPT-4o-mini experiments on EP-VQA unseen.
- CC-VQA Results: 2.9 points higher F1-score on CC-VQA unseen is achieved by the proposed method over GRPO, despite slightly lower results on the seen split.The gain comes mainly from substantially higher recall with only a slight precision decrease, indicating more errors are detected.
- CC-VQA Results: 21.4 points better technique-error recall than GRPO and SFTe is the largest recall gain reported for CC-VQA.On unseen steps, the method achieves the best recall across mistake types except temperature errors, which have few samples.
- EP-VQA Results: 5 points improvement over GRPO on EP-VQA unseen accompanies a 2.5-point improvement on the seen split.GRPO reaches 51.3 and 43.0 F1-score on the seen and unseen splits, respectively; the larger unseen gap supports stronger transfer to new steps and recipes.
- EP-VQA Results: 48.0 performance on EP-VQA unseen is 19.0 points higher than GPT-4o-mini.GPT-4o-mini outperforms base Qwen2.5 by 13.3 F1 points under the same reasoning prompt, but the adapted base model surpasses it.
- Qualitative Analysis: Qualitative examples show the method detecting subtle duration and object-state discrepancies that standard GRPO overlooks.Examples include a two-minute microwave setting instead of one minute and pouring water into a mug without a teabag.
- Ablations: The opposite reward is consistently beneficial for unseen-action generalizability, with λ = 0.2 producing the best EP-VQA performance on both splits.The accuracy gate improves F1 by 3.5 points on seen and 4.4 points on unseen data.
- Ablations: Reducing GRPO group size to G = 2 drops F1 by more than 20 points on both EP-VQA splits, motivating G = 8 despite higher computational cost for larger groups.The KL-divergence weight is fixed at β = 0.04; both lower and higher values worsen performance.
6 Conclusion
The paper introduces MD-VQA for stepwise video mistake detection across seen and unseen steps, and proposes VLM post-training with a custom reward. The approach achieves state-of-the-art performance, particularly on unseen steps.
- MD-VQA evaluates whether instructional steps were executed correctly for both seen and unseen steps.
- The proposed method is the first VLM post-training approach for video mistake detection.
- A custom reward function helps the model attend to subtle execution details that current models overlook.
- The method outperforms zero-shot, fine-tuning, and post-training baselines, especially for steps unseen during training.
A Implementation details
The experiments standardize training settings across methods where possible while adapting optimization details to each baseline’s objective and prompt configuration.
- Training setup: EP-VQA and CC-VQA training uses 800 and 802 iterations, respectively, with batch size 4 on four NVIDIA H100 GPUs.Mistake instances are upsampled by factors of 13 on EP-VQA and 3 on CC-VQA.
- Baseline configurations: GRPO matches the proposed method’s hyperparameters but removes the opposite reward, retaining only accuracy and format rewards.The reasoning prompt is used during both GRPO training and evaluation.
- Baseline configurations: SFT and SFTe use standard cross-entropy loss, while GRPO-specific parameters are omitted and remaining hyperparameters stay fixed.Training uses the base prompt; evaluation uses the base prompt for SFT and SFTe, or the reasoning prompt for SFT + R.
B Additional Results
Additional analyses track reward learning, completion length, per-error recall, and prompt designs for mistake detection. The proposed post-training approach performs strongly across mistake types, especially on an unseen utensil-error category, while unseen technique errors remain difficult.
- Post-training metrics: All format, accuracy, and opposite rewards increase during training, indicating that the policy learns to satisfy the corresponding reward criteria.The format reward rises early, whereas accuracy and opposite rewards improve gradually.
- Post-training metrics: 125 tokens is the average completion length reached toward the end of training, equivalent to approximately 90–100 words.Completion length fluctuates initially before converging.
- Per-error results: +12.5 percentage points is the proposed method’s unseen-split recall improvement for Utensil Error over the comparison methods.The method performs better than or on par with baselines across seen mistake types, while unseen Technique errors challenge all methods.
- Prompt design: Training diversifies question formulations, but evaluation uses a single fixed formulation covering video–instruction mismatches and specified mistake types.The formulations vary wording and the listed categories, while the correct execution description replaces <TEXT>.
- Prompt design: The base prompt is used by ZS, SFT, and SFTe, whereas the reasoning prompt is used by ZS+R, SFT+R, GRPO, and the proposed method.The base prompt requests a tagged YES/NO answer; the reasoning prompt additionally requests tagged reasoning and self-reflection.
D Data Cleaning and Processing
The data-processing stage addresses missing counter-descriptions and cleans inconsistent annotations so mistake-detection methods can use more reliable instruction and clip labels.
- EgoPER processing: Additional annotations provide correct-execution descriptions for mistaken EgoPER steps that originally lacked counter-descriptions.The mapping between mistake steps and correct-execution descriptions is provided in Table 9; ambiguous normal-step descriptions are also addressed.
- CaptainCook4D processing: CaptainCook4D annotations are cleaned by removing duplicated verbs, correcting grammar, and verifying that mistake-labeled clips contain mistakes.These steps target inconsistencies and artifacts in the original annotations.
E Additional Qualitative Results
Qualitative examples show that the proposed method detects instruction-specific discrepancies, including incorrect placement, approximate cooking outcomes, and ambiguous repeated actions.
- EP-VQA: The method detects that placing a tortilla on the table violates the instruction to place it on a cutting board.The cutting board is visible, and the initial instruction is followed before the final placement diverges.
- CC-VQA: For mushroom cooking, the method prioritizes softening and browning over the approximate 3–5-minute duration.The visual texture is treated as more important than exact timing.
- Evaluation scope: The qualitative evaluation covers seen and unseen splits of CC-VQA and EP-VQA.The examples are presented across both benchmark datasets and split types.
- CC-VQA: The mug-stirring example tests whether models recognize that repeated stirring satisfies the instruction to stir the contents well.The example concerns an ambiguous term and compares reasoning about the achieved result.
F.1 Results with newer model
Evaluation with Qwen3-VL-8B-Instruct shows that the newer backbone is stronger than Qwen2.5, while the proposed training strategy remains useful, especially on unseen data.
- Qualitative example: The unseen qualitative example shows standard GRPO misclassifying multiple mug stirrings as a mistake despite correct execution.The step requires stirring the mug contents well, and repeated stirring indicates correctness.
- Qwen3 evaluation: Qwen3 provides stronger zero-shot baselines than Qwen2.5 on EP-VQA, both with and without reasoning.The comparison evaluates the newer model on the same benchmark.
- Qwen3 evaluation: The proposed method improves over both Qwen3 baselines, indicating continued usefulness when starting from a more recent backbone.The task remains challenging despite the stronger model.
- Split-wise comparison: Compared with standard GRPO, the method slightly underperforms on the seen split but achieves a large improvement on the unseen split.The reported comparison differs by split.
F.2 Comparison to non-VLM methods
Additional experiments compare the method with CLIP-style models and examine the roles of chain-of-thought reasoning and textual step instructions in mistake detection.
- Open-set setting: The protocol evaluates open-set mistakes that task-specific error detection models cannot handle because test tasks differ from training tasks.The comparison therefore focuses on zero-shot CLIP-style models as the feasible non-VLM class.
- Comparison to CLIP: The method greatly outperforms CLIP on EP-VQA, indicating that reasoning capabilities are necessary for strong performance.CLIP is evaluated using text–video similarity between correct and incorrect step descriptions.
- Ablation study: GRPO with the opposite reward but without chain-of-thought predicts mistakes for almost all samples, yielding 100% error-class recall with very low precision.The result identifies a degenerate solution prevented by chain-of-thought reasoning.
- Instruction-aware detection: Removing textual step descriptions makes mistake detection often ill-defined because mistakes can be deviations from the specified instruction.The ablation evaluates the ZS + R baseline using visual information alone.