Source-linked AI summary
Toward Cognitive Supersensing in Multimodal Large Language Model
Boyi Li, Yifan Shen, Yuanzhe Liu, Yifan Xu, Jiateng Liu, Xinzhuo Li, Zhengyuan Li, Jingyuan Zhu, Yunhan Zhong, Fangzhou Lan, Jianguo Cao, James M. Rehg, Heng Ji, Ismini Lourentzou, Xu Cao
TL;DR
MLLMs remain limited on complex visual cognition because abstract spatial operations are poorly represented by text-only reasoning. The paper introduces Cognitive Supersensing, which grounds reasoning in latent visual imagery and evaluates it with CogSense-Bench; CogSense-8B outperforms state-of-the-art MLLM baselines on the cognitive tasks.
Problem
MLLMs struggle with complex visual cognition, and text-only reasoning can lose information needed for abstract visual transformations and spatial relations.
Method
Cognitive Supersensing combines an LVIP head, supervised fine-tuning, and reinforcement learning with latent rationales to align internal visual imagery with semantic reasoning.
Results
CogSense-8B achieves superior performance to state-of-the-art MLLM baselines on CogSense-Bench across five visual-cognition dimensions.
Takeaways & Limitations
Modeling the interaction between visual simulation and logical deduction is presented as an effective direction for advancing multimodal reasoning.
Takeaways & Limitations
The method assumes that the visual input contains the question image together with candidate option images.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have achieved remarkable success in open-vocabulary perceptual tasks, yet their ability to solve complex cognitive problems remains limited, especially when visual details are abstract and require visual memory. Current approaches primarily scale Chain-of-Thought (CoT) reasoning in the text space, even when language alone is insufficient for clear and structured reasoning, and largely neglect visual reasoning mechanisms analogous to the human visuospatial sketchpad and visual imagery. To mitigate this deficiency, we introduce Cognitive Supersensing, a novel training paradigm that endows MLLMs with human-like visual imagery capabilities by integrating a Latent Visual Imagery Prediction (LVIP) head that jointly learns sequences of visual cognitive latent embeddings and aligns them with the answer, thereby forming vision-based internal reasoning chains. We further introduce a reinforcement learning stage that optimizes text reasoning paths based on this grounded visual latent. To evaluate the cognitive capabilities of MLLMs, we present CogSense-Bench, a comprehensive visual question answering (VQA) benchmark assessing five cognitive dimensions. Extensive experiments demonstrate that MLLMs trained with Cognitive Supersensing significantly outperform state-of-the-art baselines on CogSense-Bench and exhibit superior generalization on out-of-domain mathematics and science VQA benchmarks, suggesting that internal visual imagery is potentially key to bridging the gap between perceptual recognition and cognitive understanding. We will open-source the CogSense-Bench and our model weights.
1 Introduction
MLLMs perform well on visual perception but struggle with cognitive operations requiring abstract visual manipulation, while text-only reasoning can lose geometric and spatial information. Cognitive Supersensing addresses this gap with latent visual imagery reasoning, a dedicated training dataset, and CogSense-Bench evaluation.
- Motivation: MLLMs recognize and describe visual content effectively but struggle with abstract layouts, future transformations, and visual-rule inference.These failures expose a gap between low-level perception and high-level cognitive reasoning.
- Motivation: Text-based Chain-of-Thought reasoning can bottleneck visual reasoning because geometric transformations and structured spatial relations are compressed into discrete tokens.The resulting information loss can make reasoning brittle for mentally rotating shapes, simulating dynamics, or inducing diagrammatic rules.
- Evaluation: CogSense-Bench evaluates visual cognition across fluid intelligence, crystallized intelligence, visuospatial cognition, mental simulation, and visual routines.Systematic evaluation finds substantial weaknesses across all five dimensions, including with Chain-of-Thought prompting.
- Approach: Cognitive Supersensing equips MLLMs with latent image-based reasoning chains through a Latent Visual Imagery Prediction head that aligns intermediate visual states with answer representations.The approach is motivated by representations that preserve geometry, continuity, and structured visual relations during problem solving.
- Approach: CogSense-Dataset provides targeted training data across five categories, while supervised fine-tuning and reinforcement learning train latent visual reasoning and optimize rollout trajectories.The contribution list identifies CogSense-8B as achieving state-of-the-art CogSense-Bench performance and strong out-of-domain generalization.
2 CogSense Dataset and Benchmark
CogSense-Dataset and CogSense-Bench target a gap in benchmarks that primarily assess semantic recognition rather than high-level visual cognition. They organize evaluation around theory-grounded cognitive mechanisms requiring multi-step reasoning and manipulation of internal visual states.
- Motivation: Existing VQA benchmarks predominantly evaluate semantic recognition or description, leaving high-level visual cognition without systematic evaluation protocols.The proposed dataset and benchmark address this gap with a unified multi-task evaluation framework.
- Categories: The benchmark assesses five theory-grounded categories: fluid intelligence, crystallized intelligence, visuospatial cognition, mental simulation, and visual routines.These dimensions correspond to visual interpretation, prediction, and reasoning capabilities associated with human cognition.
- Categories: Fluid intelligence measures solving unseen reasoning problems independently of prior knowledge, whereas crystallized intelligence measures using learned world knowledge to abstract semantic concepts from visual variation.The definitions are grounded respectively in Structure Mapping Theory, Inductive Reasoning, and Prototype Theory.
- Dataset presentation: Figure 1 presents category examples, Figure 2 presents the CogSense-Dataset distribution, and detailed statistics and pipeline information appear in the appendices.The dataset is described as a multi-task resource for evaluating visual cognition beyond perceptual recognition.
- Scope: CogSense-Dataset and CogSense-Bench probe abstract rule induction, spatial structuring, mental simulation, and attention control beyond recognition.These tasks often require composing elementary operations while maintaining and manipulating answer-oriented internal visual states.
3 Method
The method trains a multimodal model to generate textual rationales and answers while predicting answer-oriented latent visual imagery, then refines rationale sampling with reinforcement learning. It evaluates general ability alongside cognitive benchmark performance.
- Model architecture: CogSense-8B maps visual inputs and prompts into a multimodal backbone that generates reasoning rationales and answers autoregressively.Visual features are projected into the language embedding space before processing with prompt tokens.
- Training pipeline: Training uses three stages: teacher-generated reasoning chains, LVIP-augmented supervised fine-tuning, and reinforcement learning with latent rationales.Generated chains are filtered for correct answers and hallucinated content before supervised fine-tuning.
- Latent visual imagery prediction: The LVIP head predicts the latent representation of the ground-truth answer option image from pooled option-image visual-token states.Its prediction is trained against a frozen visual-encoder embedding using mean squared error.
- Supervised fine-tuning: The supervised objective combines autoregressive text loss with an LVIP mean-squared-error term, whose coefficient β balances the two objectives.At inference, the text decoder produces the final answer while the LVIP head may remain frozen for optional answer-oriented grounding.
- Latent-rationale reinforcement learning: RL samples diverse rationale trajectories using a Generative Flow Network and scores them with answer evidence plus LVIP-based representation grounding.The combined trajectory reward is R(Z; X, y) = α Rans(Z; X, y) + γ Rlvip(Z; X, y), and SubTB training is applied to accepted trajectories.
- Evaluation: The evaluation compares CogSense-8B with MLLM and VLM baselines on CogSense-Bench and reports general-task performance against the base model.The supplied table descriptions identify cognitive-ability, general-ability, and ablation evaluations.
4 Experiments
CogSense-8B is evaluated against mainstream multimodal models on cognitive and general vision-language benchmarks, with qualitative, ablation, and out-of-domain analyses. It achieves strong cognitive performance while preserving general vision-language capability and generalizing to image-based Chemistry and Math tasks.
- Cognitive Ability Results: 73.8% average accuracy makes CogSense-8B the strongest overall model on CogSense-Bench, surpassing GPT-5.2 by +33.5.
- Cognitive Ability Results: CogSense-8B narrows the human–model performance gap and outperforms the next strongest baseline across all five cognitive categories.
- Cognitive Ability Results: In qualitative pattern reasoning, CogSense-8B identifies the underlying rule clearly, whereas Gemini 2.5 Flash and GPT-5.2 produce wrong answers.
- General Ability Results: CogSense-8B maintains robust performance comparable to the base model on general vision-language benchmarks including HallusionBench, AI2D, GQA, ScienceQA, and ChartQA.
- Ablation Study: LVIP raises average accuracy to 68.0% after standard SFT, while GRPO adds +3.2 and +2.8 improvements to SFT variants without and with LVIP, respectively.
- Ablation Study: CogSense-8B reaches 73.8% average accuracy and outperforms the strongest SFT with LVIP plus GRPO baseline, indicating gains from the proposed RL method.
- Out-of-Domain Generalization: CogSense-8B improves out-of-domain EMMA performance by +6.2 on Chemistry and +8.8 on Math.
5 Related Work
The paper situates its work alongside abstract reasoning and visual cognition research, including VLM-based and latent visual reasoning approaches. It also references established visual reasoning benchmarks such as EMMA.
- The related-work discussion covers Abstract Reasoning, Visual Cognition, VLMs for Visual Reasoning, and Latent Visual Reasoning.
- Figure 5 presents sample problems from the EMMA benchmark.
6 Conclusion
The paper proposes Cognitive Supersensing with an LVIP head and SFT/RL training using Latent Rationales to align internal visual imagery with semantic reasoning. CogSense-8B outperforms state-of-the-art MLLM baselines on five-dimensional visual cognition tasks.
- Cognitive Supersensing integrates a Latent Visual Imagery Prediction head into a training paradigm for complex cognitive reasoning in MLLMs.
- The training strategy combines supervised fine-tuning and reinforcement learning with Latent Rationales to align internal visual imagery and semantic reasoning chains.
- CogSense-Bench evaluates five dimensions of visual cognition through a comprehensive visual question answering suite.
- CogSense-8B achieves superior performance to state-of-the-art MLLM baselines on the cognitive tasks evaluated.
A Statistics and Samples of Dataset and Benchmark
The appendix reports CogSense-Dataset and CogSense-Bench statistics across visual-cognition categories and describes a leakage-controlled benchmark construction process. The listed resources span multiple categories, including crystallized intelligence, visuospatial cognition, and visual routines.
- CogSense-Dataset and CogSense-Bench statistics are reported in Tables A1 and A2, respectively.
- Benchmark leakage was controlled by sampling instances proportionally from each category and removing those samples from CogSense-Dataset.
- The listed resources include Bongard-RWR+, Bongard-HOI, and other datasets under crystallized-intelligence and visuospatial-cognition categories.
- The Visual Routines category includes CVR, listed with 10K examples and an Apache-2.0 license.
B.1 Data Extraction
The data pipeline searches across visual-cognition datasets, standardizes question formats, and reformats selected problem types into multiple-choice tasks. It also uses customized prompts to generate explicit reasoning chains from short question-answer pairs.
- The pipeline searches multiple datasets covering visual-cognition tasks, including Bongard, Raven, ARC-AGI, CVR, KiVA, and STARE.
- Questions whose original metrics were not multiple-choice were reformatted into multiple-choice format.
- For Bongard problems, one positive image is mixed with negative samples as an option while remaining positive samples form the question.
- For ARC-AGI problems, ground-truth images are augmented through methods such as color modification and presented alongside the originals as options.
- Customized prompts instruct LLMs to generate explicit reasoning chains and final answers from short visual question-answer pairs.
C Human Study Design and Setup
The human study recruited 20 unpaid participants and used a stratified 100-question subset of CogSense-Bench delivered through Google Forms. The protocols received prior Institutional Review Board approval.
- Participant Recruitment: The human baseline recruited 20 participants through online platforms using random sampling without financial compensation.
- Questionnaire Design: The questionnaire contained 100 multiple-choice questions selected by stratified sampling to preserve CogSense-Bench category proportions.
- Questionnaire Design: The study was distributed digitally through Google Forms for remote administration.
- Compliance: All study protocols were reviewed and approved by the Institutional Review Board before the experiment.
D More Qualitative Examples
The qualitative examples compare visual-cognition reasoning across models, showing differing interpretations of category membership and outlier selection. CogSense-8B produces a more coherent multi-step reasoning chain than the other illustrated responses.
- Qualitative examples: Gemini 2.5 Pro instead selects C after excluding stationary vehicles, aerial stunts, and an image it considers too similar.The response prioritizes active ground riding while adding diversity through vintage style.
- Qualitative examples: Another response labels D the outlier because it shows a customized chopper on display rather than motorcycles in motion or use.This interpretation differs from the category-completion reasoning that selects a matching image.
- Qualitative examples: The examples also include a reported error occurring during question understanding.This marks question interpretation as a limitation in the qualitative evaluation.
- Qualitative examples: CogSense-8B identifies G as the best-fitting image by combining ground-based riding, motion, and stylistic consistency criteria.Its reasoning excludes static bikes, aerial stunts, and an action-consistent image with inconsistent style.
- Qualitative examples: Figure D1 presents CogSense-8B as having a coherent, multi-step logical chain that more closely matches ground truth than other models’ paths.The comparison emphasizes precision and interpretability of reasoning trajectories.