Toward Cognitive Supersensing in Multimodal Large Language Model

1 Introduction

• MLLMs can recognize and describe visual content but struggle to mentally manipulate it, while text-only CoT can bottleneck geometric and structured visual reasoning (Wei et al., 2022; Zhang et al., 2024c).The introduction gives abstract layouts, future transformations, shape rotation, dynamics simulation, and visual-rule induction as examples of difficult operations.Source: §1 · pp. 1–2
• CogSense-Bench evaluates five dimensions—fluid and crystallized intelligence, visuospatial cognition, mental simulation, and visual routines—and reveals persistent weaknesses in state-of-the-art MLLMs, including with CoT prompting.The benchmark operationalizes visual cognition beyond perceptual recognition.Source: §1 · p. 2
• Cognitive Supersensing adds an LVIP head that predicts answer-aligned visual latent sequences, then uses supervised fine-tuning and reinforcement learning to encourage coherent latent reasoning.The latent states are intended to preserve visual structure while language supplies high-level semantic guidance and final explanations.Source: §1 · p. 2
• CogSense-8B achieves state-of-the-art performance across all tasks and 73.8% overall accuracy, surpasses GPT-5.2 by +33.5, and generalizes to challenging science and mathematics VQA benchmarks.The introduction also reports outperformance of strong baselines on broader science VQA benchmarks.Source: §1 · p. 2
• CogSense-Dataset spans five categories and eleven subtasks and supports a multistage training pipeline that grounds intermediate visual latents toward the answer.Its supervised objective makes latent states predictive of the final answer’s visual representation; reinforcement learning optimizes latent-conditioned rollout trajectories.Source: §1 · p. 2, Figure 1 · p. 3, Figure 2 · p. 4

Dataset and benchmark

• CogSense-Dataset-105K and CogSense-Bench target the gap left by VQA benchmarks focused predominantly on semantic recognition or description.The dataset is described as comprehensive and large-scale, and the benchmark as a unified multi-task evaluation of high-level visual cognition.Source: §2 · p. 4

Cognitive categories

• The benchmark evaluates five complementary dimensions: Fluid Intelligence, Crystallized Intelligence, Visuospatial Cognition, Mental Simulation, and Visual Routines.These theory-grounded categories are intended to capture distinct mechanisms of human visual reasoning, including abstract reasoning, spatial structuring, simulation, and attention-driven operations.Source: §2 · p. 4
• Fluid Intelligence tests solving unseen problems, while Crystallized Intelligence assesses using learned world knowledge to abstract concepts across visual variation.Visuospatial Cognition probes reconstruction of structural relationships and decomposition of shapes; Mental Simulation infers hidden dynamics; Visual Routines assesses visual search and focused attention.Source: §2 · p. 4

Motivation

• Because CogSense tasks require multi-step reasoning and answer-oriented internal visual states, the paper proposes Cognitive Supersensing to pair rationale generation with latent visual imagery prediction.The proposed approach is intended to support the simulation-like and stateful reasoning demanded by CogSense-Bench.Source: §2 · p. 4

3.1 Preliminaries

• The three-stage pipeline synthesizes reasoning chains, jointly trains text generation and latent imagery prediction, then uses latent rationales in reinforcement learning.The model processes visual and text tokens with a shared LLM backbone, whose text decoder generates the rationale and answer while an LVIP head predicts answer-oriented latent imagery.Source: §3.1 · p. 5

3.2 Stage I: Reasoning Chain Generation

• A teacher MLLM generates step-by-step rationales and predicted answers, and chains with incorrect conclusions or hallucinated content are filtered out.The retained examples form an augmented dataset for supervised fine-tuning.Source: §3.2 · p. 5

3.3 Stage II: SFT with Latent Visual Imagery Prediction

• LVIP predicts the ground-truth option image’s latent representation from pooled hidden states for option-image tokens, jointly with supervised text generation.The target is the frozen visual encoder embedding of the correct option image; the model minimizes an MSE loss alongside autoregressive cross-entropy, and the LVIP head can remain frozen at inference for optional grounding.Source: §3.3 · pp. 5–6

3.4 Stage III: RL with Latent Rationales

• Diversity-seeking RL uses a Generative Flow Network to train rationale sampling toward a reward-induced posterior, combining answer evidence with LVIP representation grounding (Bengio et al., 2021, 2023; Lahlou et al., 2023; Zhang et al., 2022).A frozen supervised-model scorer supplies answer evidence, while a frozen LVIP head supplies the representation-level reward; flow matching trains the rationale policy rather than selecting a single deterministic chain.Source: §3.4 · pp. 6–7
• Sparse anchor scores are interpolated between rationale prefixes for SubTB training, while reference-guided fine-tuning accepts trajectories meeting a relative evidence threshold (Madan et al., 2023).At inference, the model samples multiple latent rationales and selects the answer with the highest length-normalized frozen-scorer evidence score.Source: §3.4 · pp. 7–8

4.1 Experimental Settings

• Experiments compare CogSense-8B with mainstream open- and closed-source MLLMs and VLMs using accuracy, alongside a human baseline from 20 participants answering 100 sampled CogSense-Bench questions.The human study was IRB-approved.Source: §4.1 · p. 8

4.2 Cognitive Ability Results

• CogSense-8B achieves 73.8% overall accuracy, surpasses GPT-5.2 by +33.5, and outperforms the next-strongest baseline across all cognitive categories while narrowing the human–model gap.The authors attribute the reduced gap to more effective use of visual imagery in cognitive reasoning.Source: §4.2 · pp. 8–9
• For abstract pattern reasoning, CogSense-8B identifies the underlying visual rule, while Gemini 2.5 Flash and GPT-5.2 fail to extract it and answer incorrectly.Figure 4 depicts CogSense-8B’s reasoning as a coherent multi-step chain that closely matches the ground truth.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3 General Ability Results

• CogSense-8B maintains robust performance comparable to its base model on general vision-language benchmarks, preserving foundational knowledge and instruction-following capabilities.The evaluation covers HallusionBench, AI2D, GQA, ScienceQA, RealWorldQA, ChartQA, BLINK, and MMStar.Source: §4.3 · p. 9

4.4 Ablation Study

• In ablations, LVIP raises average accuracy to 68.0%, GRPO yields gains of +3.2 and +2.8, and the full method reaches 73.8%, exceeding SFT with LVIP plus GRPO.Standard SFT nearly doubles average performance over vanilla Qwen3-VL-8B; the proposed RL method is reported as more effective than general GRPO at refining LVIP-grounded reasoning.Source: §4.4 · pp. 9–10

4.5 Out-of-Domain Evaluation

• On EMMA’s image-based Chemistry and Math subsets, CogSense-8B gains +6.2 and +8.8, respectively, supporting generalization beyond its training data.Both the questions and answer options contain images.Source: §4.5 · p. 10

5 Related Work

• The paper situates its work within abstract reasoning, visual cognition, VLMs for visual reasoning, and latent visual reasoning, with fuller discussion in Appendix E.Source: §5 · p. 10

6 Conclusion

• Table 4 presents an out-of-domain EMMA evaluation comparing CogSense-8B with its backbone; the supplied caption does not report results.Source: Table 4 · p. 10
• Cognitive Supersensing combines a Latent Visual Imagery Prediction head with SFT and RL using Latent Rationales to align internal visual imagery with semantic reasoning chains.The paradigm is designed to bridge perceptual processing and complex cognitive reasoning in MLLMs.Source: §6 · p. 10
• CogSense-8B achieves superior performance to state-of-the-art MLLM baselines on the visual-cognition tasks evaluated in CogSense-Bench.The authors argue that modeling visual simulation together with logical deduction can advance multimodal reasoning.Source: §6 · p. 10
• CogSense-Bench is introduced as an evaluation suite targeting five dimensions of visual cognition.Source: §6 · p. 10

Appendix

• The appendix lists dataset and benchmark statistics and samples, plus data-pipeline sections on extraction, reformatting, and reasoning-chain generation.Source: Appendix · p. 11
• The appendix also includes the human-study design and setup, more qualitative examples, and related works.Source: Appendix · p. 11

• To prevent leakage, instances were randomly sampled from each category at the dataset’s category proportions, then removed from CogSense-Dataset.Source: A Statistics and Samples of Dataset… · p. 12
• Table A1 lists CogSense-Dataset category descriptions and corresponding VQA-pair counts.Source: Table A1 · p. 12
• Table A2 lists CogSense-Bench category descriptions and corresponding VQA-pair counts.Source: Table A2 · p. 12

B.1 Data Extraction

• The pipeline manually selects relevant examples from diverse visual-cognition datasets across five predefined categories, then uses automated scripts for large-scale collection and cleaning.Seed sources were chosen for high relevance to downstream tasks.Source: §B.1 · p. 13

B.2 Data Reformatting

• Questions that were not originally multiple-choice are reformatted into multiple-choice format to standardize the dataset.Source: §B.2 · p. 13
• For Bongard problems, one positive image is randomly mixed with negative samples as shuffled options, while the remaining positive samples form the question.Source: §B.2 · p. 13
• For ARC-AGI problems, ground-truth images are augmented, including by color modification, and both original and augmented images are provided as options.Source: §B.2 · p. 13

B.3 Reasoning Chain Generation

• Customized prompts guide LLMs to generate explicit reasoning chains that analyze visual cues, infer rules, explain causal relationships, and provide a final answer.Prompts were tailored to different question types and transformed traditional short QA pairs into reasoning data.Source: §B.3 · p. 13
• Generated reasoning chains are filtered out when they reach an incorrect conclusion or contain hallucinated content.Source: §B.3 · p. 13

Participant Recruitment

• 20 volunteers were recruited through online platforms using random sampling, with no financial compensation or monetary incentives.The study used the participants to establish a human performance baseline.Source: C Human Study Design and Setup · p. 13

Questionnaire Design

• The questionnaire comprised 100 manually selected multiple-choice CogSense-Bench questions, stratified to preserve the benchmark’s category proportions and distributed through Google Forms.The survey was administered remotely.Source: C Human Study Design and Setup · p. 13

Compliance

• The Institutional Review Board reviewed and approved all study protocols before the study.Source: C Human Study Design and Setup · p. 13

D More Qualitative Examples

• The examples also flag question-understanding errors; Gemini identifies a displayed chopper as the outlier because it is static rather than in use.The described set otherwise shows motorcycles in motion or use.Source: D More Qualitative Examples · p. 15
• CogSense-8B selects G by excluding static displays, aerial stunts, and style-mismatched C; Figure D1 says its reasoning chain more closely matches ground truth than other models.Gemini 2.5 Pro instead selects C, emphasizing active ground riding and diversity.Source: Figure D1 · p. 15, D More Qualitative Examples · p. 15

E Related Works

• Bongard Problems test rule induction across positive and negative image sets; later datasets extend the task to real-world objects, attributes, relationships, and free-form concepts (Bongard, 1968, 1970; Nie et al., 2020; Jiang et al., 2022).The related-work discussion also cites Bongard-RWR and Bongard-RWR+ as real-world extensions (Małkiński et al., 2025; Pawlonka et al., 2025).Source: E Related Works · p. 16
• Matrix reasoning evaluates visual cognition and working memory; newer zero-shot datasets address training-heavy benchmarks, while MaRs-VQA adds rigorous human experiments and psychometric validation (Huang et al., 2023; Zhang et al., 2024b; Cao et al., 2025).Earlier benchmarks include PGM and RAVEN (Barrett et al., 2018; Zhang et al., 2019).Source: E Related Works · p. 16
• Human accuracy on the full ARC public evaluation set is 64.2%, far above current AI methods; the passage says o3 achieves negligible performance on ARC-AGI-2 (LeGris et al., 2024; Chollet et al., 2025).ARC-AGI measures skill-acquisition efficiency and out-of-domain generalization through grid-based visual program synthesis (Chollet, 2019).Source: E Related Works · p. 16
• VLMs commonly align visual encoders with pretrained language models and use instruction tuning, but static visual inputs limit dynamic search for fine-grained cues in complex images (Radford et al., 2021; Liu et al., 2023; Wu & Xie, 2024).The passage also identifies high-resolution adaptation, few-shot reasoning, compositional understanding, and constrained visual grounding as challenges.Source: E Related Works · pp. 16–17
• Latent visual reasoning uses compressed visual states, scratchpads, or latent tokens for intermediate simulation and planning, retaining visual information beyond text-centric reasoning (Zhu et al., 2025; Sun et al., 2025; Li et al., 2025a,c).Examples include Mirage and Latent Sketchpad, which generate and refine mental imagery during inference (Yang et al., 2025c; Zhang et al., 2025).Source: E Related Works · p. 17