Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?

• Existing benchmarks rarely provide closed-loop evaluation of visual details, commonsense knowledge, and logical causal reasoning in realistic scenarios.Source: §1 · pp. 1–2
• The benchmark embeds trap elements in image details while isolating them from textual information, making visual details the decisive source for answering.Source: §1 · p. 2
• It offers basic and deceptive versions, with the latter adding moral or emotional textual interference to challenge cross-modal reasoning.Source: §1 · p. 2
• Argus Inspection is a 1,430-entry benchmark covering fine-grained perception, contextualized commonsense understanding, and causal inference across varied domains and visual conditions.Source: §1 · p. 3
• The Eye of Panoptes framework uses vision-driven construction and a binary parametric Sigmoid metric to evaluate multimodal reasoning beyond rigid 0/1 scoring.Source: §1 · p. 3
• Across 26 MLLMs, results show persistent weaknesses in fine-grained detail capture and commonsense causal reasoning.Source: §1 · p. 3

2.1 Multimodal Large Language Models

• MLLMs evolved from image-text alignment and cross-modal generation toward instruction tuning, high-resolution processing, multilingual adaptability, and increasingly complex reasoning.Source: §2.1 · p. 3
• Recent multimodal reasoning paradigms integrate visual, auditory, and linguistic signals, moving MLLMs toward adaptive system-level intelligence.Source: §2.1 · p. 3

2.2 Multimodal Benchmarks

• Existing benchmarks are categorized into general vision and relational understanding, OCR-based visual comprehension, and domain-specific reasoning.Source: §2.2 · p. 3

3.1 Commonsense

• The benchmark’s commonsense taxonomy spans ten safety- and harmlessness-oriented domains, with 200 generated topics used to organize the data.Source: §3.1 · p. 3

3.2 Pattern

• Trap elements are embedded in image details and isolated from textual content so visual information becomes the sole critical source for correct responses.Source: §3.2 · pp. 3–4
• Basic and deceptive versions differ by whether moral or emotional interference is added to textual information to induce attention away from visual traps.Source: §3.2 · p. 4

3.3 Pipeline

• For each domain-topic pair, the pipeline generates a prohibition-based item and a corresponding trap action that violates it.Source: §3.3 · p. 4
• Each example contains a scenario, trap, question, and standardized answer that identifies the correct action, trap, and hypothetical bad consequence while keeping the trap textually isolated.Source: §3.3 · p. 4
• Images are generated and curated, then text-image pairs undergo multimodal verification and human filtering, yielding 1,430 retained entries.Source: §3.3 · p. 4

4.1 Metrics

• Traditional 0/1 scoring is insufficient because Argus evaluates both trap-element recognition and alignment with the standard action.The two dimensions respectively inform perception and reasoning assessment, so omitting either produces an incomplete evaluation.Source: §4.1 · p. 4
• A LLM-based scorer evaluates whether trap elements are mentioned (x-score) and whether the action aligns with the given standard answer (y-score), assigning rule-based scores from 1–4.The scorer used is GPT-4o-2024-11-20.Source: §4.1 · pp. 4–5
• The binary Sigmoid function non-uniformly maps intermediate x- and y-score combinations to [3.372e-06, 0.9734], rather than treating all responses as simply correct or incorrect.Parameters p, q, and r control steepness, y-score weighting, and offset; the selected setting is (p=3, q=0.8, r=6).Source: §4.1 · p. 5
• The indicator function assigns a d-score of 1 only when a model’s image description identifies the trap elements, separating visual recognition from subsequent reasoning.A model lacking the required image recognition capability receives 0 for this indicator.Source: §4.1 · p. 5

4.2 Framework

• The framework combines image-centered data construction with binary parametric Sigmoid smoothing and a visual-recognition indicator to assess fine-grained perception and causal reasoning.It is designed for multimodal reasoning with particular emphasis on visual details.Source: §4.2 · p. 6
• The framework’s binary metric also accounts for inconsistent identity and consistent anomaly, and can guide response evaluation and model post-training.These cases cover responses that identify hazards without choosing the standard action or choose the correct action without mentioning the trap elements.Source: §4.1 · p. 4, §4.2 · p. 6

5 Experiments

• GPT-4.1-2025-04-14, Gemini-2.5-Flash-Preview-05-20, and Gemini-2.5-Pro-Preview-05-06 occupy the top three positions, with both Gemini versions scoring above 0.40.Overall scores are weighted across domains by domain volume.Source: §5 · p. 6
• Closed-source GPT and Gemini models rank higher overall than open-source LLaMA and DeepSeek models.Source: §5 · p. 6
• Across ten domains, deceptive-level evaluation is significantly harder than basic-level evaluation, although rare cases show deceptive scores exceeding basic scores by only 0.01 to 0.02.The authors hypothesize that longer descriptive text may sometimes guide models toward the image.Source: §5 · p. 6
• o4-mini-2025-04-16 is most susceptible to deception, whereas the Gemini-2.5 series shows resistance to deceptive challenges.Source: §5 · p. 6
• Current MLLMs still lack strong fine-grained capture and commonsense causal reasoning capabilities.The paper identifies this as a continuing challenge for human-centered applications and embodied intelligence.Source: §5 · p. 6

6 Conclusions

• Argus Inspection and Eye of Panoptes are introduced to evaluate visual fine-grained reasoning, commonsense understanding, and causal reasoning in MLLMs.The conclusion reports a highest score of only 0.46 across 26 mainstream MLLMs.Source: §6 · p. 6

A Taxonomy of Commonsense

• The commonsense taxonomy contains 10 domains, 200 topics, and 1,430 data entries, with some content overlapping multiple domains.Source: A Taxonomy of Commonsense · p. 9

B.1 d-score

• The d-score measures whether the target model’s image description identifies trap entities or visually similar elements before inference.It is binary: failure receives 0 and successful mention receives 1, without requiring exact status, location, or attributes.Source: §B.1 · p. 10

B.2 x-score

• The x-score measures whether a response finds and analyzes the trap, assigning 4 points for analysis-based decisions and 1 point when the trap is not found or mentioned.Intermediate scores distinguish brief analysis from mere mention.Source: §B.2 · p. 10

B.3 y-score

• The y-score measures agreement with the standard answer, distinguishing exact reasoning consistency, shared purpose with differing details, same choice with different basis, and opposite choices.The scale ranges from 4 points for fully consistent reasons to 1 point for completely opposite suggestions.Source: §B.3 · p. 10, §B.2 · p. 10

C.1 Learderboard on Arugs-basic

• Argus-basic evaluation covers 26 MLLMs, with results presented in a leaderboard ordered by descending overall score.Source: §C.1 · p. 11, Table 4 · p. 11

• The evaluation covers experimental results from 26 MLLMs on the Argus-deceptive version.Source: §C.2 · p. 11
• The supplied passages identify the leaderboard and its ordering but do not report individual model scores.Source: §C.2 · p. 11, Table 5 · p. 11
• Table 5 presents the Argus-deceptive leaderboard with overall scores listed in descending order.Source: Table 5 · p. 11, §C.2 · p. 11

• The analysis uses t-tests to compare the means of the two Argus versions and assess statistical significance.Source: §C.3 · p. 12
• All domains show statistically significant differences between Argus-basic and Argus-deceptive, with p-values far below 0.05.The corresponding t-test significance indicators are marked “True.”Source: §C.3 · p. 12, Table 6 · p. 12
• Cohen’s d indicates relatively large effect sizes in domains 02, 04, and 06.These domains therefore show more pronounced practical differences between the two Argus versions.Source: §C.3 · p. 12

• Most of the 26 MLLMs have p-values below 0.05, indicating statistically significant differences among models.Source: §C.4 · p. 12, Table 7 · p. 12
• o4-mini-2025-04-16 and DeepSeek-VL-2-20241213 show relatively high Cohen’s d values, indicating substantial differences.Source: §C.4 · p. 12
• Seed-1.5-VL-250428 and Gemini-2.5-Pro-Preview-05-06 have higher p-values, so their differences do not reach statistical significance.Source: §C.4 · p. 12