Source-linked AI summary
BabyVision: Visual Reasoning Beyond Language
Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Y. charles, Yiping Bao, Yuantao Fan, Guopeng Li, Haiyang Shen, Xuanzhong Chen, Wendong Xu, Shuzheng Si, Zefan Cai, Wenhao Chai, Ziqi Huang, Fangfu Liu, Tianyu Liu, Baobao Chang, Ming Wu, Xiaobo Hu, Kaiyuan Chen, Yixin Ren, Yang Liu, Yuan Gong, Kuan Li
TL;DR
Contemporary MLLMs perform strongly on knowledge-intensive benchmarks but remain weak on foundational visual tasks that humans acquire early, motivating a language-independent evaluation. BabyVision benchmarks these abilities across diverse tasks, while BabyVision-Gen extends evaluation to visual generation; leading models show a substantial human–model gap.
Problem
Contemporary MLLMs succeed on knowledge-intensive evaluations yet show consistent weaknesses on basic visual tasks that children solve with little difficulty.
Method
BabyVision evaluates early-vision abilities with 388 questions across 22 subclasses and four domains, while BabyVision-Gen evaluates visual reasoning through generated visual outputs.
Results
49.7% overall accuracy for the best-performing model versus 94.1% for adult humans, a 44.4% absolute gap across all four domains.
Takeaways & Limitations
The findings indicate that strong language-heavy benchmark performance does not imply robust visual foundations, and position BabyVision as a diagnostic tool for grounded visual intelligence.
Takeaways & Limitations
MLLMs struggle with mental affine transformations because language provides an insufficient coordinate system for preserving volumetric constraints.
Abstract
from arXiv · showhide
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and reasoning capabilities. We also explore solving visual reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.
1 Introduction
BABYVISION targets early visual abilities that humans develop before language, addressing weaknesses that remain hidden by knowledge-intensive MLLM benchmarks. Across its benchmark and generative extension, the paper finds a substantial human–model gap and explores visual-output evaluation.
- Motivation and benchmark: BABYVISION contains 388 questions across 22 subclasses and four domains targeting atomic visual skills while minimizing reliance on linguistic knowledge.Its domains are Fine-grained Discrimination, Visual Tracking, Spatial Perception, and Visual Pattern Recognition.
- Evaluation results: 49.7% overall accuracy for the best-performing model trails adult humans at 94.1% by 44.4 percentage points, with deficits across all four domains.The largest failures occur in Visual Tracking and Spatial Perception.
- Evaluation results: BABYVISION exposes foundational visual failures not captured by existing benchmarks, despite strong high-level multimodal reasoning.Reported errors include losing curve identity through intersections and failing to mentally transform 3D structures.
- Generative evaluation: BABYVISION-GEN evaluates visual reasoning through image generation because humans often solve tracking, pattern, and spatial tasks by drawing rather than verbalizing.The paper also introduces an automatic evaluation tool for generated visual solutions.
- Generative evaluation: Frontier image and video generators show promising gains on visual tracking and fine-grained discrimination, although reliability and solution consistency remain limited.The reported experiments include Nano-Banana-Pro and Sora-2.
2 Related Work
Prior multimodal benchmarks emphasize expert knowledge and semantic reasoning, while related studies document persistent perceptual weaknesses and language leakage. Developmental evidence motivates evaluating vision-centric abilities that emerge before language and can be demonstrated nonverbally.
- Knowledge-intensive benchmarks: Existing benchmarks such as MMMU and HLE primarily target college-level or expert-level multimodal reasoning across many disciplines.Mathematical and domain-specific benchmarks likewise emphasize knowledge-intensive problem solving.
- Perceptual limitations: BLINK reports a human score of 95.7% and reveals a substantial human–MLLM gap on classical vision problems requiring no specialized knowledge.MMStar and MMVP further report language leakage and failures on clear perceptual differences.
- Developmental motivation: Developmental research shows that infants acquire object permanence, occlusion tracking, depth, and shape discrimination before language.These findings support treating core visual competencies as independent of linguistic mediation.
- Developmental motivation: BABYVISION selects tasks solvable by young children and minimizes verbal reasoning demands to compare models with visual development across childhood.The design also draws on evidence that children externalize visual reasoning through drawing before verbalizing solutions.
3 BabyVision
BABYVISION is a four-category benchmark of foundational visual skills built through a multi-stage curation process. Its questions are filtered, reviewed, and formatted to reduce language-based shortcuts and ensure visually grounded answers.
- Benchmark design: BABYVISION divides vision-centric reasoning into Fine-grained Discrimination, Visual Tracking, Spatial Perception, and Visual Pattern Recognition across 22 subtypes.The categories cover subtle differences, trajectories, 3D structures, relationships, and logical or geometric patterns.
- Data curation: The curation pipeline comprises taxonomy and seed selection, data collection and filtering, and annotation with quality assurance.Figure 4 presents these three stages as the benchmark construction process.
- Quality control: Images relying heavily on text or cultural knowledge are discarded, and each retained question must have an unambiguous answer approved by two independent experts.Persistent disagreements lead to revision and eventual removal.
- Dataset statistics: 388 questions are paired with unique images, including 135 multiple-choice and 253 fill-in-the-blank items with balanced answer distributions.Questions average 25.9 words and are tightly grounded in visual content.
- Evaluation protocol: A unified prompt template encourages reasoning before the final answer, while Qwen3-Max evaluates semantic equivalence between model outputs and ground truth.The judge is reported to show 100% consistency with human evaluators.
4 BabyVision-Gen
BABYVISION-GEN reformulates selected benchmark tasks as visual-output problems, asking models to annotate images with solutions rather than answer in language. Its automatic judge closely matches human judgments, enabling scalable evaluation of generated visual reasoning.
- Task formulation: BABYVISION-GEN adapts a subset of questions by instructing models to generate images that trace paths, complete patterns, or mark spatial relationships.This tests visual solutions that may exceed a model’s ability to verbalize answers.
- Task formulation: BABYVISION-GEN comprises 280 questions across 21 subtypes in the same four categories as BABYVISION.One subtype is excluded because it does not naturally support a generative solution format.
- Inference and evaluation: Models are instructed to add minimal overlays such as circles, lines, arrows, or text while preserving the original image context.The output makes the predicted visual answer explicit without replacing the input scene.
- Inference and evaluation: The automatic judge compares the input, human-annotated solution, and generated output, returning a binary subtype-specific correctness decision.Criteria include identical selected options and matching traced routes for mazes.
- Validation: 96.1% agreement with human judgments supports automatic evaluation, based on 269 of 280 instances and F1=0.924.The human evaluation used PhD-level annotators assessing all Nano-Banana-Pro outputs.
5 Main Experiments
BabyVision evaluations show that current MLLMs remain far below humans on foundational visual competencies, with weaknesses spanning fine-grained discrimination, tracking, spatial perception, and pattern recognition. Scaling and explicit reasoning help inconsistently, while visual generation offers limited gains on some tasks but struggles with spatial coherence.
- Overall Performance: 49.7% overall accuracy for Gemini3-Pro-Preview trails the 94.1% adult human baseline by 44.4 percentage points, with deficits across all four domains.Most frontier MLLMs also perform below the average 3-year-old, while Gemini3-Pro-Preview still lags typical 6-year-olds by about 20 points.
- Fine-grained Analysis: Best subtype scores remain low on foundational tasks, including Count 3D Blocks at 20.5% and Find the Same at 26.5%.These failures are associated with structured scene representations such as depth-aware composition and object permanence rather than superficial recognition.
- Model Scaling and Reasoning: 22.2% overall accuracy for the best open-source model remains far below the top proprietary system, although Thinking variants outperform same-scale Instruct variants.At 235B parameters, Qwen3VL Thinking scores 22.2% versus 19.5% for Instruct; scaling generally helps but is not monotonic because 4B-Thinking reaches 14.6% versus 13.1% for 8B-Thinking.
- Fine-grained Analysis: Visual Tracking exposes identity-maintenance failures: most models score near zero on tracing curves through intersections and frequently switch tracks at crossings.Gemini3-Pro-Preview reaches 83.3% on Lines Observation, but most other models remain near zero.
- Fine-grained Analysis: Pattern-based tasks are comparatively stronger when transformations are discrete and rule-based, reaching 67.9% on Logic Patterns and 70.0% on Rotation Patterns.Overlay Patterns and Mirroring Patterns remain challenging, with best scores of 50.0%.
- BabyVision-Gen: 18.3% overall accuracy for NanoBanana-Pro leads visual generation models, but Visual Tracking reaches only 6.7% and Maze and Connect the Lines score 0% across all models.Generation models perform relatively better on Fine-grained Discrimination and Visual Pattern Recognition, while overall reliability and consistency remain limited.
6 Discussion
The discussion attributes MLLM failures on basic visual tasks to information loss from language-mediated reasoning and examines whether reinforcement learning or visual generation can address this gap.
- Why Do Frontier Models Fail on Seemingly Simple Tasks?: Four failure modes expose a visual modality gap: fine-grained detail, manifold identity, spatial imagination, and visual pattern induction.These failures are presented as systematic consequences of compressing perceptual reasoning into language.
- Why Do Frontier Models Fail on Seemingly Simple Tasks?: Language-mediated reasoning flattens continuous geometric structure, making subtle boundaries, curves, and spatial relations difficult to preserve.The paper contrasts this lossy compression with humans’ direct shape matching and contour integration.
- Why Do Frontier Models Fail on Seemingly Simple Tasks?: Visual tracking fails when models lack persistent curve representations, causing track switching or hallucinated endpoints at intersections.The discussion links this failure to combinatorial branching when overlapping trajectories are converted into discrete instructions.
- Why Do Frontier Models Fail on Seemingly Simple Tasks?: Spatial imagination fails because descriptive logic provides an insufficient coordinate system for volumetric constraints and hidden 3D structures.Models may therefore infer impossible projections or hallucinate unseen structures.
- Insight from Training: Can RLVR Help Visual Reasoning?: +4.8 points overall accuracy followed RLVR fine-tuning, but visual tracking showed little to negative improvement.The authors associate this limitation with tracking’s low amenability to verbalization and RLVR’s emphasis on structured thinking-token reasoning.
- Beyond Language: Visual Externalization: Visual generation is proposed as a way to bypass verbalization, but current generated solutions still contain noticeable errors.The paper concludes that generation requires robust visual understanding rather than operating as a sufficient substitute for it.
7 Conclusion
BabyVision measures early-vision abilities beyond language and finds that frontier MLLMs remain far behind humans across core visual domains. The paper attributes these deficits to a verbalization bottleneck and proposes visual generation as an additional evaluation direction.
- Conclusion: 49.7% for Gemini3-Pro-Preview versus 94.1% for humans demonstrates a large gap across fine-grained discrimination, tracking, spatial perception, and pattern recognition.The reported deficits are consistent across all four domains.
- Conclusion: BabyVision identifies four failure modes that stem from verbalization discarding visual information that cannot be faithfully expressed in language.The named failures are loss of fine-grained detail, loss of manifold identity, spatial imagination failure, and appearance-structure confusion.
- Conclusion: BABYVISION-GEN extends evaluation to visual outputs, with preliminary evidence that generation models can exhibit human-like visual thinking.The conclusion presents visual-fidelity-preserving architectures as a likely direction for progress.
A Evaluation Details for BabyVision
BabyVision evaluation uses a judge model to compare model outputs with ground-truth answers for semantic equivalence. The evaluator returns a binary True-or-False judgment under explicit matching rules.
- Evaluation Procedure: Qwen3-Max serves as the judge model for evaluating semantic equivalence between model outputs and ground-truth answers.The judge receives the question, correct answer, and another model’s output.
- Evaluation Criteria: Matching requires equivalent facts, entities, and key details, while incorrect, incomplete, or meaning-changing outputs do not match.Small wording differences, formatting differences, and synonyms are ignored.
- Output Format: The evaluator’s final output is restricted to True or False with no additional explanation.The reasoning process is specified as internal to the judge.
B Evaluation Details for BabyVision-Gen
BabyVision-Gen evaluates visual reasoning by asking generation models to annotate the original image with their solution. The prompt preserves the source image while restricting outputs to precise overlays.
- Generation Evaluation: Visual generation models annotate the original image so they can demonstrate solutions through visual marks while preserving the original context.The approach targets visual reasoning beyond language output.
- Annotation Constraints: Allowed annotations are circles, lines, arrows, or text labels added on top of the unchanged image.The prompt prohibits modifying, redrawing, or altering original visual content.
- Annotation Constraints: Annotations must use bright visible colors and remain minimal and precise, marking only what the question requests.These constraints are intended to make the visual answer clearly distinguishable while retaining the source image.
B.2 Automatic Evaluation Protocol
The protocol evaluates generated visual solutions by comparing them with human-annotated ground truth using subtype-specific criteria and exact-match decisions. BABYVISION-GEN organizes these criteria across multiple visual task types.
- Evaluation procedure: Gemini-3-Flash compares the input, human-annotated ground truth, and model-generated images to judge solution correctness.The judge returns a binary decision based on subtype-specific criteria.
- Decision rules: The protocol marks a generation true only when its answer matches the ground truth, disregarding stylistic differences.Different marked elements, routes, numbers, characters, or missing answers are false.
- Subtype criteria: Subtype-specific rules require exact agreement for selected options, marked elements, counted values, characters, paths, and spatial-reasoning answers.Examples include identical circled options, exact numbers, and the same traced or maze route.
- Task coverage: The protocol covers discrimination, pattern completion, counting, path tracing, character recognition, spatial reasoning, and visual pattern identification tasks.These criteria correspond to the listed BABYVISION-GEN subtypes and categories.
- Dataset organization: Table 4 reports the distribution of questions across BABYVISION-GEN categories and subtypes.One BABYVISION subtype is excluded because it does not naturally support a generative solution format.
B.5 Human Evaluation and Validation
The authors validate automatic evaluation against independent human judgments on generated outputs. The automatic scorer shows high agreement with human evaluation, while the reported classification metrics quantify its performance.
- Human evaluation: Human annotators independently judged each generated image against the ground truth.The evaluation used PhD-level annotators and compared their judgments with the automatic scorer.
- Agreement: 96.1% agreement was achieved between automatic and human judgments across 280 NanoBanana-Pro outputs.The agreement covers 269 of 280 samples.
- Evaluation metrics: The evaluation reports 0.957 precision, 0.893 recall, and a 0.924 F1 score for automatic assessment.These metrics summarize whether the automatic evaluator identifies cases judged correct by humans.
- Confusion matrix: Table 5 presents the confusion matrix comparing automatic and human evaluation, with green cells denoting agreement and red cells disagreement.The table provides the error breakdown underlying the reported agreement.
C Additional Experimental Results
Additional experiments report Qwen3VL performance across BabyVision question types and compare original models with GRPO variants by subtype.
- Qwen3VL results: Table 6 reports Qwen3VL Instruct and Thinking performance using Avg@3 across BabyVision question types.Values average Pass@1 accuracy across three random runs and include standard deviations.
- GRPO results: Table 7 reports GRPO results by subtype and highlights the larger value between the original and GRPO models.The table separates category, subtype, and metric groups.