MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacity

1 Introduction

• Existing benchmarks either lean toward textual reasoning or lack systematic classification and in-depth analysis of vision-based cognitive behaviors.The section contrasts these gaps with MME-CC’s vision-based task organization.Source: §1 · pp. 1–2
• MME-CC addresses gaps in cognitive-capacity benchmarks by organizing 11 visual reasoning tasks across spatial, geometric, and visual-knowledge dimensions.The benchmark uses 1,173 expert-annotated questions to analyze 16 representative MLLMs.Source: §1 · p. 2
• Chain-of-Thought reasoning typically follows extraction, reasoning, and verification, with visual extraction involved throughout.This pattern is reported alongside the benchmark’s analysis of MLLM cognitive capacity.Source: §1 · p. 2

2.1 Data Construction Pipeline

• The data pipeline defines and pilots tasks, collects and verifies images, standardizes them, then filters overly simple, redundant, or ambiguous items.The construction process includes a 10-person annotation team with dedicated subtask leads and a task lead.Source: Figure 2 · p. 3, §2.1 · pp. 3–4

2 MME-CC

• MME-CC organizes 11 subtasks into Spatial, Geometric, and Visual Knowledge Reasoning categories.Table 2 reports each subtask’s data source, sample size, and average input and output token lengths.Source: §2 · p. 3, Table 2 · p. 4, §2.3 · p. 5

2.2 Data Quality Assurance

• Each sample receives at least two review rounds, with disagreements escalated to a subtask lead and periodic audits by a dedicated QA team.Samples yielding zero accuracy across screened models are randomly reviewed for annotation issues or hidden shortcuts, then corrected or removed if problematic.Source: §2.2 · p. 5

2.3 Benchmark Statistics

• MME-CC emphasizes vision-based reasoning because its text contains no task-specific solution information, so solving relies primarily on visual understanding.The benchmark comparison describes vision-based status as task information derived from images rather than text.Source: §2.3 · p. 5, Table 3 · p. 5

3.2 Main Results

• Gemini-2.5-Pro reaches 42.66% overall accuracy, while its Geometric and Spatial Reasoning scores are both below 30%.Its Visual Knowledge Reasoning score is 74.63%.Source: Table 4 · p. 6, §3.2 · p. 6

3.1 Experimental Setup

• The evaluation covers 16 MLLMs across Spatial, Geometric, and Visual Knowledge Reasoning, using official API defaults for proprietary models and unified decoding settings for open-source models.DeepSeek-V3-0324 judges answers against gold references; manual checks of 99 items achieved 95% agreement with human judgments.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 Main Results

• Reasoning-oriented models consistently outperform non-reasoning counterparts on Spatial and Geometric Reasoning tasks.Within the GPT series, GPT-5 (high) outperforms GPT-4.1 on both dimensions.Source: §3.2 · p. 6
• Qwen2.5 performance improves consistently as model size increases from 7B to 32B and 72B.The authors state that complex visual perception and reasoning require broader knowledge capacity, while smaller models face performance limitations.Source: §3.2 · pp. 6–7

4.1 RQ1: Models excel at different tasks, yet overall performance remains unsatisfactory

• GPT-5 (high) leads Spatial Reasoning at 30.3%, while Gemini-2.5-Pro leads Visual Knowledge Reasoning at 70.7%.Doubao’s Geometric Reasoning advantage is localized to Jigsaw Puzzle, while Gemini-2.5-Pro is more robust across other geometric challenges.Source: §4.1 · p. 7

4.2 RQ2: MLLMs remain far from “thinking like humans”

• The analyzed chain of thought follows three stages: problem understanding and information collection, core analysis and reasoning, then conclusion formation and verification.The overall structure remains stable although task-specific tactics vary.Source: §4.2 · pp. 7–8
• Visual extraction recurs throughout reasoning; in Satellite Image Matching, the model revisits images to recheck building orientation and layout.Those checks can prompt revisions to earlier spatial judgments.Source: §4.2 · p. 8
• Most models score 20%-30% on spatial and geometric tasks, and none exceeds 2% in Maze; excessive verification can stall and repeat.The authors hypothesize that long chains dilute attention and obscure visual details, degrading outcomes.Source: §4.2 · p. 8
• Text-first prompting improves most tasks; the reported ablation excludes tasks with instruction conflicts, including Chart Modification and Visual Knowledge Reasoning.The authors attribute gains mainly to better textual alignment rather than stronger intrinsic visual reasoning.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 RQ3: MLLMs exhibit recurring failures in orientation judgment, entity consistency, and instruction following

• Recurring errors include orientation and reference-frame mismatches, cross-view identity failures that cause double counting or omission, and ignoring counterfactual constraints.For non-literal instructions, models may prioritize literal visual content over the stated constraint.Source: §4.3 · pp. 8–9

5 Related Work

• General multimodal benchmarks cover image understanding, multidisciplinary reasoning, and open-domain evaluation, including MMBench, MMMU [24], and RealWorldQA.Other cited examples include SEED-Bench, MathVista [11], MMMU-Pro [25], OlympiadBench [6], and VisualWebBench.Source: §5 · p. 9
• Some benchmark items permit reliance on textual cues, format priors, or OCR, making visual reasoning difficult to isolate.The passage identifies this issue across general multimodal benchmarks.Source: §5 · p. 9
• Language-independent evaluations and visual puzzles target visual inference, but many tasks use synthetic data, repetitive templates, or narrow formats.The cited examples include ZeroBench [21] and VisuLogic [23].Source: §5 · pp. 9–10

6 Conclusion

• MME-CC organizes eleven tasks into spatial, geometric, and visual-knowledge dimensions and evaluates sixteen representative models.Gemini-2.5-Pro scores 42.66 overall versus 30.45 for GLM-4.5V; spatial and geometric reasoning are both at or below 30%.Source: §6 · p. 10
• The conclusion identifies orientation confusion, limited cross-view identity persistence, and reduced counterfactual-instruction adherence as recurring errors.It also reports a three-stage CoT pattern—extract, reason, verify—with visual extraction throughout, and consistent gains from first verbalizing visual content.Source: §6 · p. 10
• The authors position MME-CC’s task- and dimension-level diagnostics as actionable for evaluation and model design.They expect the analyses to inform training signals and architectures that better couple visual perception with structured reasoning.Source: §6 · p. 10

A.1.1 Satellite Image Matching

• Satellite matching pairs a landmark map with two nearby Street View queries and seven candidate locations, requiring a distinct location for each query.Candidates are checked for topological validity and visual confusability; viewpoint consistency is verified, and blind guessing both answers has accuracy of about 2.4%.Source: §A.1.1 · p. 13

A.1.2 Indoor Directional Reasoning

• Indoor Directional Reasoning uses adjacent VR-tour views to infer a query view’s cardinal orientation from an oriented anchor view.Controls remove directional cues, verify continuity and floorplan consistency, and balance room types and directions.Source: §A.1.2 · p. 13

A.1.3 Indoor Deduplication Counting

• Indoor Deduplication Counting asks models to count unique instances of two defined object categories across views from one apartment tour.Inclusion rules, examples, and manual identity-consistency review address ambiguity and cross-view duplication.Source: §A.1.3 · p. 13

A.2.1 Gomoku Variation

• Gomoku Variation presents hybrid-piece endgames curated to have one winning move, with duplicate or symmetric boards removed.Uniqueness is checked manually and programmatically, while standardized coordinates and formatting suppress shortcuts.Source: §A.2.1 · pp. 13–14

A.2.2 Unblock Me

• Unblock Me asks for both the minimum move count and ordered moves on levels whose unique minimal solutions are independently verified.Human annotators and the game solver verify solutions; ambiguous levels and UI hints are removed.Source: §A.2.2 · p. 14

A.2.3 Maze

• Maze items ask for digits on the shortest entrance-to-exit path, retaining only mazes with a verified unique shortest route.Dual human tracing and algorithmic checks validate the route; revealing arrows, traces, and artifacts are removed.Source: §A.2.3 · p. 14

A.2.4 Jigsaw Puzzle

• Jigsaw Puzzle asks for a one-to-one mapping from removed pieces to labeled empty slots, excluding symmetric or ambiguous matches.Physical test-fitting validates mappings, and image normalization reduces lighting and shadow cues.Source: §A.2.4 · p. 14

A.3.1 Sandbagging

• Sandbagging requires exactly one correct response among four ordered sub-questions, while Counterfactual Instruction applies a specified inversion to unambiguous image facts.For Sandbagging, the correct-answer position is randomized and output constraints are checked automatically; counterfactual items use deterministic inversion maps and verifiable answers.Source: §A.3.1 · p. 14, §A.3.2 · pp. 14–15

B Data Quality Assurance Protocol

• Most tasks use a two-stage QC process: leads pilot-review a sample, codify pitfalls into guidelines, and use those rules for full-scale validation.The stated goal is to ensure items are correct, unambiguous, and sufficiently difficult to distinguish model capabilities.Source: B Data Quality Assurance Protocol · p. 15
• Indoor Directional Reasoning and Unblock Me receive 100% validation by senior task leads because their cognitive demands resist simple rule-based checking.This expert-only protocol is paired with scalable validation elsewhere in the benchmark.Source: B Data Quality Assurance Protocol · p. 15

Satellite Image Matching

• Satellite matching QC removes auxiliary interface clues and rejects ambiguous landmarks with symmetrical features or unclear orientation.The task’s QC rules aim to preserve a precise, visually grounded location judgment.Source: Satellite Image Matching · pp. 15–16

Indoor Deduplication Counting

• Deduplication QC requires enough overlapping views to make counting nontrivial and defines potentially ambiguous object categories in advance.Examples include whether an empty flowerpot counts as pottery and whether lounge chairs differ from dining chairs.Source: Indoor Deduplication Counting · p. 16

Gomoku Variation

• Gomoku QC accepts all valid optimal moves when a position has multiple equally good solutions, rather than forcing one ground-truth move.The task’s examples include a defensive block and an offensive setup as potentially optimal alternatives.Source: Gomoku Variation · p. 16

Indoor Directional Reasoning

• Directional reasoning items receive expert-only review and require precise cardinal, relative, or coordinate-based descriptions instead of vague directional language.Unblock Me combines canonical-solution programmatic checks with senior manual review of the puzzle’s visual clarity and state.Source: Indoor Directional Reasoning · p. 16, Unblock Me · p. 16

C LLM-as-a-Judge Prompts

• The judge prompt for Unblock Me awards credit only when both the minimum steps and the set of blocks to move match the reference.For other open-ended answers, the general scoring guidance compares final answers with references, allows mathematically equivalent forms, and requires all subparts to be correct.Source: C LLM-as-a-Judge Prompts · pp. 17–18

• Figures 5–14 illustrate representative Doubao-1.6-Pro-Vision failures across spatial, geometric, and instruction-dependent MME-CC tasks.The cases cover cross-view grounding, spatial planning and constraints, and instruction following; they complement the broader error-pattern analysis in Section 4.3.Source: D Details of Error Cases · pp. 18–19

E.1 Incorrect Orientation and Reference Frame Alignment

• Satellite matching overlooks geometric cues for superficial textures, while directional reasoning defaults to linguistic heuristics instead of propagating constraints across rooms.Both examples concern incorrect orientation or reference-frame alignment.Source: §E.1 · p. 19

E.2 Lack of Cross-View Object Identity Persistence

• Indoor Deduplication Counting produces omissions or redundant counts when the model fails to preserve object identity across views.Source: §E.2 · p. 19

E.3 Insufficient Spatial Planning and Constraint Simulation

• Maze, Gomoku, Jigsaw, and Unblock Me examples show local path choices, misread positions, color-based matching, and failures to simulate feasible rearrangements.These cases illustrate insufficient spatial planning and constraint simulation.Source: §E.3 · p. 19

E.4 Over-Reliance on Literal Descriptions in Instruction-Conditioned Reasoning

• Sandbagging and counterfactual examples show failures to follow constrained outputs or answer in the counterfactual world; Chart Modification outputs full tables despite logical response constraints.Source: §E.4 · p. 19

F Ablation experiment setup and analysis

• Table 5 reports results for the eleven subtasks, formatting each entry as the ablation score, base accuracy, and their difference.The ablation adds an instruction to describe relevant image content before answering; the supplied passage does not provide the table's values.Source: F Ablation experiment setup and analysis · p. 19
• Error-case figures cover Satellite Image Matching, Indoor Directional Reasoning, Indoor Deduplication Counting, and Maze.These are identified as Figures 5–8, respectively.Source: Figure 5 · p. 20, Figure 6 · p. 21, Figure 7 · p. 22
• Additional error-case figures cover Gomoku Variation, Jigsaw Puzzle, Unblock Me, Sandbagging, and Counterfactual Instruction.Gomoku Variation has two captions, Figures 9 and 10; the other tasks appear in Figures 11–14.Source: Figure 9 · p. 24, Figure 10 · p. 25, Figure 11 · p. 26