PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

Yinghui Li, Jiayi Kuang, Peng Xing, Daixian Liu, Yongheng Zhang, Junnan Dong, Shu-Yu Guo, Yangning Li, Qingyu Zhou, Wenhao Jiang, Hai-Tao Zheng, Ying Shen, Liang Lin, Philip S. Yu

arXiv:2603.18472v2cs.AIcs.CV

TL;DR

The paper addresses whether MLLMs genuinely understand discrete visual symbols, a gap left by systems and evaluations centered on natural images and broad reasoning. It introduces a five-domain benchmark organized by three cognitive levels and finds a recurring recognition-reasoning inversion: models often underperform on basic symbol recognition while appearing competent on more complex reasoning. The findings indicate that linguistic priors, templates, and procedural inference can compensate for incomplete visual grounding, motivating evaluations and training that keep reasoning anchored to discrete visual evidence.

  • Problem

    The central gap is limited evidence about whether MLLMs genuinely recognize discrete symbols rather than merely producing plausible language-driven answers.

  • Method

    The paper constructs a unified benchmark spanning five symbolic domains and three cognitive levels from perception to reasoning and critical symbolic judgment.

  • Results

    Models frequently show a recognition-reasoning inversion, underperforming on foundational perception while appearing relatively competent on higher-level tasks through compensatory inference.

  • Takeaways & Limitations

    The results support evaluations and training objectives that explicitly separate grounded recognition from compensatory inference and keep reasoning anchored to visual evidence.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains unclear. We present a multi-domain benchmark spanning language, culture, mathematics, physics and chemistry, organized into three cognitive levels: perception and recognition, combination and reasoning, and association and critical thinking. Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures. These results show that symbolic understanding remains a major bottleneck for multimodal intelligence and motivate training and evaluation schemes that prioritize grounded perception in discrete semantic spaces.

Introduction

The paper frames discrete-symbol understanding as a distinct multimodal challenge because meaning depends on precise visual units and their structured relations. It introduces a unified, hierarchical benchmark to test recognition, reasoning, and contextual symbolic judgment across five domains.

  • Discrete symbols differ from natural-scene inputs because a single local visual error can change the meaning of a character, graph, circuit, or chemical structure.Mainstream MLLMs are primarily trained to map continuous visual signals to holistic semantic narratives.
  • Prior evaluation suites and specialized symbolic methods remain fragmented and rarely distinguish genuine symbol recognition from downstream manipulation.The paper identifies this separation as an unresolved limitation across existing multimodal evaluation.
  • The benchmark spans language, culture, mathematics, physics, and chemistry across perception and recognition, combination and reasoning, and association and critical thinking.This hierarchy follows a progression from local visual units to compositional integration and contextual semantic judgment.
  • The benchmark contains more than 13,000 image-question-answer instances and 38 sub-tasks, emphasizing diagnosis of visual grounding rather than competition alone.Its design tests whether models rely on visual evidence or on linguistic priors, memorized templates, and shortcuts.
  • Bringing social-scientific and natural-scientific symbols together tests whether MLLMs have general symbolic capacity or only domain-dependent success shaped by pre-training patterns.Language and culture stress convention and context, while mathematics, physics, and chemistry stress formal structure and low tolerance for local error.

Results

Across symbolic domains, models often struggle with foundational visual recognition while appearing more capable on higher-level reasoning. The results attribute this inversion to language-dominant compensation that can conceal weak visual grounding and destabilize downstream symbolic consistency.

  • Language symbols: Language-symbol tasks reveal forced normalization and unstable localization: models repair anomalous glyphs or confuse legal and illegal characters before reliable character-space representations form.These failures extend into misspelled-character detection and visual-semantic correction, where early perceptual errors destabilize later outputs.
  • Cultural symbols: Cultural-symbol performance is stronger for familiar emoji-to-word mappings but drops when interpretation requires idiomatic composition, negative operators, discourse constraints, or cultural convention.Character-level overlap can remain higher than exact idiom recovery, indicating a gap between local emoji meaning and culturally grounded interpretation.
  • Mathematical symbols: Mathematical-symbol tasks show a recognition-reasoning inversion: models miss graph or geometric details at Level 1 yet sometimes use linguistic rules and familiar templates to solve higher-level tasks.Qualitative cases show reasoning by elimination or symbolic computation replacing direct visual judgment.
  • Physical symbols: Physics models often recite relevant laws while failing to map formulas, graphs, circuit relations, and diagram structures into correct symbolic representations.Early visual errors such as misread constants, graphs, or omitted relations can derail otherwise coherent multi-step reasoning.

Discussion

Across five domains, MLLMs often underperform on foundational symbol perception while appearing stronger on later reasoning, revealing a recognition-reasoning inversion. This mismatch can conceal incorrect visual parses and motivates evaluations and training that keep reasoning grounded in symbolic visual evidence.

  • Recognition-reasoning inversion: Across five domains, MLLMs frequently underperform on perceptual baselines but appear to recover on reasoning tasks through linguistic priors, procedural templates, or contextual inference.The authors identify this reversal as the operational form of cognitive mismatch.
  • Recognition-reasoning inversion: Models can display strong semantic flexibility and sometimes repair early perceptual errors through downstream inference, making correct end-task answers unreliable evidence of genuine visual understanding.This compensation can fail when symbolic precision cannot be approximated by linguistic expectations.
  • Model design: The findings suggest that current training privileges vision-to-language concept mapping over stable symbolic visual primitives, requiring stronger discrete-symbol supervision and tighter perceptual grounding.The proposed direction includes objectives and architectural biases that preserve fine-grained topological and local structural information.
  • Human comparison: Unlike models, human participants perform strongly on foundational perceptual tasks and decline as association, correction, and multi-step reasoning become more demanding.Model failures often originate in early perception and are later concealed by fluent reasoning, whereas human failures often occur after perception.
  • Evaluation: Separating recognition, reasoning, and critical symbolic judgment across domains can expose cases where prior knowledge, semantic plausibility, or pattern completion produces correct answers despite incorrect visual parses.The benchmark is intended to provide a clearer target for systems that genuinely perceive and manipulate discrete symbolic structures.

Methods

The benchmark evaluates discrete semantic understanding across five domains using a three-level cognitive hierarchy, domain-specific tasks, and symbolic-aware metrics. It combines more than 13,000 instances, multiple closed- and open-source MLLMs, and a 1,000-instance human baseline to diagnose visual grounding separately from downstream reasoning.

  • Benchmark design: The benchmark spans language, culture, mathematics, physics, and chemistry through perception and recognition, combination and reasoning, and association and critical thinking levels.The hierarchy progresses from individual symbolic units to compositional integration and contextual semantic judgment.
  • Benchmark design: The hierarchy distinguishes visual grounding from downstream reasoning because small local errors in discrete symbolic images can alter the global answer.Task design emphasizes precision, compositional structure, and cross-level diagnosis rather than only end-to-end success.
  • Task construction: Language tasks assess handwritten Chinese character errors and correction, cultural tasks use emoji sequences for lexical and idiomatic inference, and mathematical tasks include function graphs and geometric figures.These tasks target visual-semantic and compositional abilities across symbolic domains.
  • Task construction: More than 13,000 meticulously curated instances combine reannotated existing resources with newly constructed datasets to measure symbolic fidelity as cognitive demands increase.Additional construction details are provided in Supplementary Sec. C.2.
  • Models and baseline: The evaluation covers nine closed- and open-source MLLMs with task-specific direct prompts, representing current multimodal systems.The model set includes GPT-4o, Claude-sonnet-4, Gemini-2.5-pro, o3, Qwen-max, Qwen2.5-VL, InternVL3-8B, Deepseek-vl2-tiny, and LLaMA3-LLaVA-Next-8B.
  • Models and baseline: A human baseline uses a stratified sample of 1,000 instances completed by five highly educated bilingual annotators under the same prompts.The comparison tests whether model difficulty trajectories resemble human symbolic cognition.
  • Evaluation: Metrics are domain-specific and distinguish partial recognition, approximate semantic recovery, and exact symbolic correctness.Language evaluation includes precision, recall, F1, exact match, and edit distance, while cultural evaluation includes exact-match, overlap, and semantic-similarity measures.

A Extended Introduction

The paper frames discrete-symbol understanding as a mismatch between continuous visual training and the exact recognition, composition, and consistency required by symbolic images. Its benchmark reveals weak foundational recognition, heuristic compensation, and domain-dependent performance across MLLMs.

  • Motivation: Current MLLMs are optimized for continuous visual signals and semantic narratives, whereas discrete-symbol images require exact recognition and structured composition.A single misread stroke, bond, or relation can change the intended meaning.
  • Benchmark framework: The benchmark spans five symbolic domains and three cognitive levels, progressing from perception and recognition to combination and reasoning, then association and critical thinking.It is designed to evaluate symbolic capabilities in structured, abstract visual representations rather than ordinary natural-image understanding.
  • Cross-domain results: No single model performs consistently across symbolic domains, while language symbols are especially difficult and mathematics and chemistry are relatively stronger.The benchmark-level comparison reports broader coverage for proprietary models than open-source counterparts.
  • Core finding: Models often perform better on higher-level reasoning than foundational recognition, indicating reliance on linguistic priors, memorized patterns, or procedural imitation.Strong language reasoning can partially mask deficient visual perception through contextual inference.
  • Language-symbol analysis: Language-task failures include forced normalization, inaccurate anomaly detection, weak contextual integration, and severe semantic drift during correction.In faked-character detection, most models scored below 2 F1; Qwen2.5-vl’s performance was comparable to much larger GPT-4o, suggesting scale alone does not determine results.
  • Language-symbol analysis: Only Gemini-2.5-pro and InternVL3-8B achieved double-digit exact-match scores for visual-semantic character correction, while LLaMA3-llava-next-8b scored zero.Low edit distance for the stronger models indicated directionally relevant corrections, whereas several others showed severe character- and semantic-level deviation.

B.2.1 Models perform well at the low level but still face challenges

English word and idiom tasks show comparatively stronger performance when models can combine recognized visual elements into familiar semantic compositions. However, emoji decoding remains vulnerable to hallucination when surrounding symbols impose a conflicting constraint.

  • English recognition: GPT-4o achieved F1 scores of 55.8 and 35.2 on English word-level and sentence-level word-and-idiom recognition, respectively.The passage attributes this relative strength to greater exposure to similar English text during training.
  • English recognition: A model correctly combined pine and apple semantics to infer the compound word pineapple.This case illustrates successful semantic composition and association from multiple visual elements.
  • Emoji interpretation: Emoji decoding can ignore a prohibition symbol and retrieve a semantically opposite idiom from the meanings of individual emojis.The example produced “bright idea” instead of the correct “Not the brightest bulb.”

B.2.2 Models show limited performance in Chinese idiom tasks

Chinese idiom tasks expose limits in multi-character symbolic association: models perform poorly on longer idioms despite better performance when translating individual emojis.

  • Chinese idiom tasks: GPT-4o achieved accuracy scores of 3.3 and 5.0 on four-character and multi-character idiom tasks.Even Qwen2.5-VL lagged behind GPT-4o on these more challenging tasks.
  • Chinese idiom tasks: Models often reason from superficial emoji attributes while missing object semantics and homophonic relationships needed for the correct idiom.The passage gives “bucket” as a homophone for the character meaning “same” as an example of the required reasoning.
  • Chinese idiom tasks: Higher Chr-1 accuracy than longer-idiom accuracy indicates stronger individual-emoji translation than contextual visual intuitive semiosis.The longer tasks additionally require cross-modal divergent thinking and phonetic-semantic associative reasoning.

B.2.3 Semantic Similarity Analysis Reveals Random Guessing Patterns in Association and Critical Thinking

Models show a recurring mismatch: basic visual recognition is often weaker than higher-level reasoning, with language-based procedures compensating for limited image use.

  • Most models perform worse on basic recognition than on complex reasoning tasks, contrary to intuitive cognitive expectations.
  • 37.3 points on function type classification was achieved by Qwen2.5 despite higher-level tasks permitting compensation through linguistic logic.Level 1 requires precise visual localization, whereas Levels 2 and 3 emphasize reasoning and rule generalization.
  • Models may infer a quadratic function by eliminating alternatives from textual definitions despite limited pixel-level perception.In Case 8, the upward-opening U-shaped curve supported this reasoning-based inference.
  • Model adaptability varies: smaller models can perform clearly defined reasoning through language matching, while larger models better handle open-ended error detection.
  • Models often bypass explicit visual information, reconstructing graph answers through intersection coordinates, slope calculation, and algebraic reasoning.This longer symbolic procedure replaced direct visual judgment in Case 9.

B.3.3 Fine-grained task analysis

The benchmark analysis attributes weak foundational recognition partly to training objectives that favor fluent vision-to-language mapping over fine-grained visual localization.

  • Most multimodal training encourages “jump mapping” from visual inputs to language concepts rather than fine-grained visual localization.VQA and captioning prioritize coherent descriptions, leaving models poorly equipped for detailed symbolic tasks.
  • Language dominance may suppress visual-modality development, making Level 1 weakness partly a consequence of selective reliance on language reasoning.

B.4.1 Limited recognition of sparse physical symbols across most models

Physical-symbol tasks expose weak visual-to-symbolic mapping: models can recite physical laws yet misread symbols, relations, and diagram structure, while a few models perform more reliably.

  • 14.1% accuracy for GPT-4o, 16.9% for Qwen2.5-vl, and 1.8% for LLaMA3-llava-next-8b were reported on Level 1 physics-symbol tasks.Most models had mean accuracies below 30%.
  • Models often state physical-law descriptions correctly but fail to map them into symbolic equations and graphs.One model represented power as linearly rather than quadratically related to current.
  • A logically sound reasoning path still produced an incorrect conclusion after the model changed H = 3mg/k to H = mg/k.The numerical-stage parameter error disrupted the final answer.
  • Deepseek-vl2-tiny showed opaque, unstable reasoning, suggesting some correct answers may reflect template retrieval or pattern matching rather than systematic deduction.
  • Gemini-2.5-pro achieved 60.2% accuracy on electrical symbol recognition, indicating stronger parsing of low-density, partially overlapping formula symbols.
  • Mechanical diagram consistency correction was a common Level 3 bottleneck, with many models nearly failing while Gemini-2.5-pro, o3, and Claude-sonnet-4 performed relatively acceptably.
  • Models may identify circuit components through letter labels yet confuse the rheostat slider P with power and make incorrect deductions.

B.5.1 Most models show limited capability in basic chemical symbol recognition

Chemical tasks reveal limited recognition of sparse structural notation, although some models achieve stronger reasoning or higher-level performance by using memorized patterns and domain knowledge.

  • Gemini-2.5-pro achieved 46.1% on element identification, 26.7% on chemical-bond recognition, and a 39.4% mean score at Level 1.Its mean score exceeded those of other models.
  • Models recognize explicit atom labels but often ignore skeletal-formula rules and count atoms through memorized templates.
  • Models mechanically combined benzene-ring and nitro-group formulas to obtain 6 hydrogen atoms instead of the correct 5 after substitution.
  • Sparse lines, intersecting connections, abbreviations, and aromatic-ring conventions cause substantial bond-counting and structural-recognition errors.
  • o3 performed best at Level 2, despite Gemini-2.5-pro having stronger symbol recognition.
  • Models can detect coefficient anomalies in chemical equations but often propose incorrect corrections, relying on pattern matching rather than atom conservation.
  • Gemini-2.5-pro achieved 86.7 in task 7 reaction product prediction at Level 3.
  • Some models use chemical knowledge heuristics, such as low-temperature conditions for lithium-reagent reactions, to filter options and reach correct answers.

C.2.3 Data Validation

The benchmark uses automated checks and domain-specific human review to validate data integrity, readability, correctness, and task annotations across symbolic domains.

  • Automated validation detects duplicate or missing entries and verifies image-file integrity and readability.
  • Human experts assess linguistic and emoji samples for natural usage and remove nonstandard or inappropriate content.
  • Mathematics, physics, and chemistry reviewers verify problem statements, answers, task assignments, and annotated error types.
  • Approximately 96.7% of the data was retained after validation.

C.2.4 Data Statistics and Analysis

The benchmark was statistically balanced across five symbolic domains and three cognitive levels, then evaluated with domain- and task-appropriate metrics and representative MLLM baselines.

  • Dataset composition: Statistical analysis checked for underrepresented subtasks and triggered additional collection or self-instruct augmentation where needed.
  • Dataset composition: 13,148 samples comprise the final benchmark across five symbolic domains and three hierarchical cognitive levels.
  • Dataset composition: 3,388, 5,848, and 3,912 samples correspond to Perception and Recognition, Combination and Reasoning, and Association and Critical Thinking, respectively.
  • Dataset composition: The five domains contain 1,838 Language, 2,986 Culture, 2,935 Mathematics, 1,715 Physics, and 3,674 Chemistry samples.
  • Evaluation: The study benchmarks representative open-source and closed-source MLLMs spanning multiple parameter scales and multimodal architectures.
  • Evaluation: Evaluation uses domain-specific metrics, including F1 and character-count measures for detection, exact match and edit distance for correction, and BLEU-1/BLEU-2 for idiomatic generation.
  • Evaluation: Some models infer individual emoji meanings yet fail to produce the correct idiom, yielding semantically related but lexically or structurally different outputs.

D Extended Conclusion and Future Perspectives

The study finds that MLLMs can reason about visual symbols while often failing at foundational symbol grounding, revealing a recognition–reasoning mismatch. It concludes that symbolic understanding remains an underexplored frontier and outlines training, architectural, benchmark, and cognitive-science directions.

  • The benchmark disentangles perceptual recognition, compositional reasoning, and critical symbolic cognition in discrete semantic spaces.
  • Current MLLMs frequently fail at foundational visual symbol grounding despite impressive reasoning capabilities.
  • The recognition-reasoning inversion means higher-level reasoning performance can mask deficiencies in low-level symbolic perception.
  • Future directions include symbolic-primitive supervision, structured perceptual bottlenecks, tighter vision–reasoning integration, interactive benchmarks, and cognitive-science-informed curricula.
  • Symbolic understanding remains a critical, underexplored frontier for multimodal intelligence.

E Additional Case Studies

The case studies examine discrete-symbol understanding across language, physics, and chemistry, revealing failures in precise recognition and grounded reasoning. Models may invoke relevant knowledge yet still produce incorrect or superficially plausible answers.

  • Scope: The supplementary cases span language, physics, and chemistry tasks from character detection to reaction-condition inference.They examine visual-symbol processing across foundational recognition, reasoning, and domain-specific analysis.
  • Language cases: Models misclassified correct characters as anomalies because they lacked precise symbolic discrimination.In the character-detection example, the model altered a correct character while preserving other sentence content.
  • Language cases: Misspelled-character detection requires comparing source and correct sentences while ignoring punctuation, markers, and unknown characters.The task operationalizes character-level comparison rather than unconstrained sentence generation.
  • Physics cases: In physics, models could list relevant formulas but failed to combine them with problem-specific conditions, producing physically incorrect answers.The reasoning remained at formula stacking and sometimes ended with a seemingly plausible result.
  • Chemistry case: The chemistry case shows a model identifying key reactants and invoking internal knowledge while evaluating competing reaction conditions.The example contrasts candidate temperatures and durations for the same reactants and product.
Loading 2603.18472v2…