PaperlayerView exact revision on arXiv

Source-linked AI summary

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

Jinzhe Li, Gengxu Li, Jinnan Li, Yuan Wu, Yi Chang

arXiv:2608.29286v1cs.AI

TL;DR

Existing MLLM evaluations largely assume valid inputs and underassess autonomous error processing. MMPCBench introduces a multimodal benchmark with a fine-grained taxonomy and hierarchical evaluation, finding systematic proactive-critique weaknesses, especially for visual errors, alongside a consistency gap between reasoning and final responses.

  • Problem

    Existing evaluations largely assume complete, accurate, and unambiguous inputs, while multimodal proactive error processing remains insufficiently assessed.

  • Method

    MMPCBench combines four primary error types spanning 12 subcategories with hierarchical evaluation of detection, diagnosis, remediation, and reasoning–response consistency.

  • Results

    14 MLLMs show systematic weaknesses in proactive critique, particularly for subtle visual anomalies, while reasoning models often suppress internally detected errors in final responses.

  • Takeaways & Limitations

    The findings support shifting multimodal systems toward autonomous input validation rather than prioritizing instruction-following over input scrutiny.

  • Takeaways & Limitations

    The benchmark covers only English and Chinese text inputs, limiting generalizability across linguistic families.

Abstract

from arXiv · show

As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.

1 Introduction

MMPCBench addresses the gap between ideal-input evaluation and real-world multimodal queries containing flaws, ambiguities, or contradictions. It benchmarks proactive error processing and finds weak performance, a reasoning–response consistency gap, and divergent remediation strategies across 14 MLLMs.

  • Existing MLLM evaluations largely assume complete, accurate, and unambiguous user inputs, unlike real-world deployment.
  • Proactive Critique requires models to detect invalid premises or ambiguities without prompting and respond through correction, clarification, or related remediation.
  • Existing frameworks leave gaps by emphasizing ideal-input accuracy, text-only evaluation, refusal strategies, or detection without systematic remediation.
  • MMPCBench introduces a multimodal benchmark with 12 fine-grained error types and a hierarchical protocol for detection, diagnosis, and remedial utility.
  • 14 MLLMs show limited proactive-critique proficiency, especially on subtle visual anomalies, while reasoning models often suppress internally detected errors in final answers.

2 Related Work

Prior work studies erroneous instructions, unanswerable visual questions, text-image contradictions, and explicit error detection, but MMPCBench extends evaluation toward holistic multimodal proactive reliability.

  • Text-focused benchmarks establish baselines for detecting erroneous instructions and misleading premises, while multimodal benchmarks examine unanswerable visual questions and text-image contradictions.
  • Many existing studies narrow their scope to limited response behaviors rather than comprehensive proactive remediation.
  • ProcessBench, DeltaBench, QuestBench, and ErrorRadar evaluate explicit error identification within reasoning processes, where models are instructed to act as verifiers.
  • MMPCBench broadens this landscape through a 12-subcategory taxonomy, visual attacks, response-strategy analysis, and internal–final-answer consistency evaluation.

3 Task Formulation and Error Taxonomy

MMPCBench represents each MLLM input as textual context, visual information, and a query, then classifies defects across four primary error types and 12 subcategories.

  • An MLLM input is modeled as I = {T, V, Q}, where textual and visual information provide premises and the query defines the task goal.
  • Expression Error covers ambiguity in textual visual-premise descriptions and unclear intent in incomplete or overly abstract instructions.
  • Premise Contradiction includes cross-modal, inter-visual, and intra-textual conflicts among the supplied premises.
  • Missing Premise covers absent or damaged essential information, including light and medium visual loss, missing images, missing instructions, and semantic mismatch.
  • Beyond Capability identifies queries whose answers cannot be logically inferred from the available textual and visual evidence.

4 Benchmark Construction

MMPCBench is constructed by sampling existing data, injecting targeted multimodal and textual defects, and filtering candidates through model verification and human review. This process yields 3,146 high-quality unanswerable instances from 12k synthesized candidates.

  • The benchmark construction pipeline consists of data sampling, error injection, and two-step filtration using model-based verification and human review.
  • Source questions span nine disciplines and are modified to differ from their original benchmarks while preserving their underlying subject matter.
  • Defect injection uses model rewriting, gradient-guided visual degradation, generative image editing, few-shot textual rewriting, and structural component manipulation.
  • 12k candidate samples were synthesized with category-specific injection strategies and prompt templates.
  • 4,832 candidates passed Stage 1 model checks, and human review produced 3,146 final instances with Cohen’s κ = 0.85.

5 Evaluation Framework and Metrics

The framework evaluates proactive critique hierarchically: models must detect erroneous inputs, diagnose their root causes, and provide useful remedies. These stages are aggregated into PCQ and assessed through an LLM-as-a-Judge protocol whose reliability is supported by human-alignment analysis.

  • Step I: Error Detection Accuracy: EDA measures whether a model explicitly identifies an input as unreasonable or erroneous across the evaluation set.Compliance, hallucinated affirmative responses, and silent corrections without acknowledgment count as missed detections.
  • Step II: Diagnostic Precision: Diagnostic Precision evaluates root-cause explanations only on successfully detected samples, scoring them as precise, relevant, or irrelevant.The scores are 2, 1, and 0, respectively, according to whether the explanation identifies the core defect.
  • Step III: Strategic Effectiveness: Strategic Effectiveness evaluates remedial utility on detected samples through correction, clarification, and refusal strategies.Responses receive scores from 0 to 2 based on the explicitness and actionability of the remedy.
  • Composite Evaluation: PCQ integrates global detection with diagnosis and strategy quality as a holistic measure of proactive critique performance.Diagnostic Precision and Strategic Effectiveness are normalized to [0, 1] before integration.
  • Automated Evaluation and Reliability: The protocol uses three advanced-model judges with metric-specific aggregation, while reliability checks report Fleiss’ κ = 0.70 and Human–LLM agreement κ = 0.71 with 78% accuracy.EDA uses majority voting, DP averages all judges, and SE combines majority-selected strategies with aligned-judge scores.

6 Experiment

Evaluation across 14 MLLMs shows that proactive critique is weak, with performance driven mainly by error detection and strongly shaped by model-family training paradigms. Reasoning models also lose many valid critiques between internal reasoning and final responses.

  • Overall performance: 26.1 average PCQ and 33.2% average EDA show that models fail to produce critical responses in roughly two-thirds of erroneous-input scenarios.The benchmark measures EDA, DP, SE, and composite PCQ across the evaluated models.
  • Overall performance: Qwen3-VL leads, proprietary Claude, Gemini-3, and GPT models form a middle tier, and Gemma-3, GLM-4.6V, and Doubao-Seed-1.6-vision occupy the bottom tier.The distribution forms a three-tier hierarchy across model families.
  • Scaling and family effects: PCQ gaps stay below 10 points within model families but exceed 30 points across families, indicating stronger effects from shared training paradigms than model scale.Within Qwen3-VL, PCQ rises from 38.7 at 8B to 47.4 at 235B, whereas Gemma-3 stagnates or slightly regresses.
  • Scaling and family effects: EDA dominates cross-model PCQ variability, with CV = 0.43 and r = 0.97, compared with DP CV = 0.16 and SE CV = 0.13.The quality term Q = N(DP)+N(SE) has only weak correlation with PCQ, at r = 0.39.
  • Behavioral patterns: GPT-5.1 (high) and Claude-Sonnet-4.5 (Thinking) achieve stronger critique quality but conservative EDA, while Qwen3-VL gains PCQ through higher EDA despite lower DP.GPT-5.1 (high) records DP 1.79 and SE 1.95 with EDA 31.7%; Qwen3-VL records EDA 53.3%–60.0% with DP 1.36–1.47.
  • Error-category performance: PCQ averages 12.9 for Expression Error, 21.0 for Premise Contradiction, 26.4 for Missing Premise, and 65.4 for Beyond Capability.Beyond Capability errors are more discoverable because they directly obstruct task completion, whereas subtle defects require proactive scrutiny.
  • Error-category performance: Visual-only errors generally remain below 25% PCQ, while cross-modal errors show the largest intermodel divergence and require joint text, visual, and consistency reasoning.Light-to-Severe scores do not follow a universal monotonic pattern.

7 Conclusion

MMPCBench evaluates proactive critique across 3,146 samples, 12 fine-grained error types, and 14 MLLMs. The study finds systematic weakness, a reasoning-to-response consistency gap, visual-only blind spots, and divergent Solver–Consultant strategies.

  • Benchmark and findings: MMPCBench evaluates Proactive Critique across 3,146 samples and 12 fine-grained error types using 14 MLLMs.The benchmark is presented as a framework for evaluating proactive reliability in multimodal systems.
  • Benchmark and findings: The benchmark reveals systematic proactive-critique weakness driven by sycophancy that prioritizes instruction-following over input scrutiny.The conclusion frames this as a central finding of the evaluation.
  • Benchmark and findings: Reasoning models often detect errors internally but suppress those findings in final responses, while visual-only errors remain a universal blind spot.The study also observes a behavioral split between aggressive Solvers and cautious Consultants.
  • Implication: The findings motivate a shift toward autonomous input validation for more honest and robust multimodal systems.This is the conclusion’s stated direction for future system design.

Limitation

The benchmark’s main stated scope limitation is language coverage: it uses only English and Chinese text inputs, limiting generalization across linguistic families. The dataset otherwise spans multiple error types, subjects, and multimodal complexities.

  • Scope limitation: The benchmark is limited to English and Chinese text inputs, so findings may not fully generalize across linguistic families.The authors identify multilingual expansion as future work.
  • Dataset coverage: The dataset contains 3,146 instances across four primary error types, with Missing Premise as the largest category at 1,405 instances.Several contradiction subtypes contain approximately 290 instances each.
  • Dataset coverage: Samples span 9 academic and professional subjects, question lengths from a few tokens to over 4,000 tokens, and 0 to 5 images.Multi-image subsets additionally require inter-visual reasoning.

C Error Injection and Construction Details

MMPCBench constructs multimodal error samples through targeted rewriting, image manipulation, modality removal, and mismatched-image replacement. The benchmark is evaluated with hierarchical metrics and bootstrap-based significance analysis.

  • Textual Error Injection: LLM-based rewriting injects five textual error types while preserving the original image.The targeted types are Ambiguity, Unclear Intent, Cross-Modal Contradiction, Intra-Textual Contradiction, and Beyond Capability.
  • Visual Error Injection: Inter-Visual Contradiction pairs an original image with a generated image containing contradictory evidence for a necessary visual premise.An LLM first generates an editing instruction, after which Gemini-3-Pro-Image synthesizes the second image.
  • Visual Error Injection: Gradient-guided degradation obscures salient visual regions at Light, Medium, and Severe levels using 20–30%, 40–50%, and 70–80% degradation proportions.The operations include blurring, pixelation, and geometric warping.
  • Modality and Mismatch Injection: Complete modality removal creates Missing Image and Missing Instruction samples by retaining only text or only images.Neither construction requires LLM involvement.
  • Modality and Mismatch Injection: Text-Image Mismatch replaces original images with semantically unrelated images while preserving the number of images and question structure.The replacement images come from a different subject domain in the dataset pool.
  • Statistical Evaluation: The evaluation uses 95% bootstrap confidence intervals from 1,000 resamplings of 3,146 samples, with adjacent-rank significance tested by paired bootstrap p-values.Most adjacent-rank gaps are significant; Qwen3-VL-235B-A22B-Instruct and Claude-Sonnet-4.5 (Thinking) have non-overlapping intervals.

F.2 Human Validation of the Evaluation Protocol

The evaluation protocol is supported by automated and human agreement analyses, while additional tests examine fine-grained model behavior, prompt sensitivity, reasoning consistency, and false alarms on valid inputs.

  • Validation: The three-judge automated panel achieves Fleiss’ κ = 0.70 across the full benchmark, indicating substantial inter-judge agreement.The judges are GPT-5.2, Gemini-3-Pro, and Claude-Sonnet-4.
  • Validation: On 180 stratified samples, human–LLM agreement reaches overall κ = 0.71 with 78% accuracy.Per-metric Cohen’s κ values are 0.77 for EDA, 0.71 for DP, and 0.67 for SE, with accuracies of 88%, 75%, and 73%.
  • Fine-Grained Results: Inter-Visual Contradiction and Missing Visual Premise remain difficult, whereas identifying an error is generally easier than diagnosing complex cross-modal inconsistencies.Diagnostic precision is relatively higher for Beyond Capability and Mismatch.
  • Robustness Analyses: Prompt sensitivity is limited: the maximum shifts are 2.7 EDA points and 2.9 PCQ points, with model rankings unchanged across prompt styles.The comparison uses original, formal, and concise prompts on a stratified 300-sample subset.
  • Reasoning Consistency: For Claude-Sonnet-4.5 (Thinking), increasing reasoning effort raises EDAR from 47.7% to 53.7% and EDAF from 39.3% to 46.7%, while DDR falls from 27.0% to 19.0%.Increasing the maximum token budget at high effort does not produce monotonic gains, and DQD remains close to zero.
  • Valid-Input Robustness: Valid-input analysis finds URR-V never exceeds 0.16%, with no model producing more than one unfounded full refusal among 639 valid inputs.Higher error recall can coincide with more false alarms, while conservative models obtain higher precision.

G.5 Judge Strategy Bias and Model-Family Robustness

The judge analyses find little evidence of systematic strategy or model-family bias, while dual-error testing shows that models more often detect that an input is problematic than identify every interacting defect.

  • Judge Strategy Bias: Clarification and Refusal receive nearly identical mean Strategic Effectiveness scores across all three judges.The differences are 0.03, 0.01, and 0.05 for GPT-5.2, Gemini-3-Pro, and Claude-Sonnet-4, respectively.
  • Model-Family Robustness: Leave-family-out judge rankings remain highly consistent with the original ranking, with Spearman ρ = 0.969 and Kendall τ = 0.912.Several response models receive their highest reported DP from the Gemini judge rather than a same-family judge.
  • Dual-Error Evaluation: On 50 dual-error samples, Qwen3-VL models reach MEDA values of 74%–76%, while the best Type Coverage is 46% and the best All Detected rate is 27%.The subset covers three interacting-error combinations and evaluates detection and diagnostic coverage.
  • Dual-Error Evaluation: Models more often recognize that a compound input is problematic than identify all interacting defects.The gap between MEDA and All Detected captures incomplete coverage of multiple ground-truth error types.

H Qualitative Cases

Qualitative cases illustrate failures and successes across ambiguity, capability, contradiction, and missing-information errors. The examples emphasize that visual or textual inconsistencies can make questions unanswerable unless the model identifies and addresses the faulty premise.

  • Representative Cases: GPT-5.2 fails to detect ambiguity, while GPT-5-mini successfully requests missing information for a Beyond Capability sample.Claude-Sonnet-4.5 (Thinking) internally identifies a Cross-Modal Contradiction but omits it from the final response.
  • Textual Contradictions: An Intra-Textual Contradiction makes a policy question unanswerable because it defines government spending as both increasing and decreasing.The conflicting premises receive equal status, so the intended direction must be clarified.
  • Missing Premises: Missing visual premises prevent answering when a referenced figure omits arch geometry, loads, and hinge positions needed for structural calculations.The omitted information blocks support-reaction and axial-load computation.
  • Cross-Modal Contradictions: A Cross-Modal Contradiction occurs when text says point B is lower than E although the image shows B ≈1.6mg/ml and E ≈1.2mg/ml.The question does not specify whether the text or image takes precedence.
  • Visual Mismatch: A mismatched skull-base image cannot answer a question asking for structure 205 because the image and labels belong to an unrelated subject.This is a visual mismatch rather than a failure of the requested anatomical identification alone.
  • Ambiguity and Missing Information: Ambiguous or incomplete instructions can remain unresolved even when the underlying task is specified, as in the valuation question lacking a clear task and dividend data.The missing information leaves open whether to calculate a required return or intrinsic value.
Loading 2608.29286v1…