GENIUS: Generative Fluid Intelligence Evaluation Suite

Ruichuan An; Sihan Yang; Ziyu Guo; Wei Dai; Zijun Shen; Haodong Li; Renrui Zhang; Xinyu Wei; Guopeng Li; Wenshan Wu; Wentao Zhang

Source-linked AI summary

GENIUS: Generative Fluid Intelligence Evaluation Suite

Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, Wentao Zhang

arXiv:2602.11144v1Feb 11, 2026cs.LGcs.AIcs.CV

TL;DR

Existing benchmarks largely test crystallized knowledge rather than fluid visual reasoning, so GENIUS evaluates pattern induction, constraint execution, and contextual adaptation; 12-model results reveal substantial deficits and gains from training-free intervention.

Problem
Existing UMM benchmarks primarily assess crystallized knowledge, while evidence on their ability to perform novel, context-grounded visual reasoning remains limited.
Method
GENIUS formalizes GFI into three dimensions and evaluates them through five expert-designed tasks using rule compliance, visual consistency, and aesthetic quality metrics.
Results
Evaluations of 12 models reveal substantial GFI deficits; context-guiding interventions improve performance, while models often understand task intent but fail to produce compliant visuals.
Takeaways & Limitations
GENIUS provides a standard for measuring fluid visual intelligence, and its training-free attention adjustment consistently improves performance across tasks.

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) have shown remarkable progress in visual generation. Yet, existing benchmarks predominantly assess $\textit{Crystallized Intelligence}$, which relies on recalling accumulated knowledge and learned schemas. This focus overlooks $\textit{Generative Fluid Intelligence (GFI)}$: the capacity to induce patterns, reason through constraints, and adapt to novel scenarios on the fly. To rigorously assess this capability, we introduce $\textbf{GENIUS}$ ($\textbf{GEN}$ Fluid $\textbf{I}$ntelligence Eval$\textbf{U}$ation $\textbf{S}$uite). We formalize $\textit{GFI}$ as a synthesis of three primitives. These include $\textit{Inducing Implicit Patterns}$ (e.g., inferring personalized visual preferences), $\textit{Executing Ad-hoc Constraints}$ (e.g., visualizing abstract metaphors), and $\textit{Adapting to Contextual Knowledge}$ (e.g., simulating counter-intuitive physics). Collectively, these primitives challenge models to solve problems grounded entirely in the immediate context. Our systematic evaluation of 12 representative models reveals significant performance deficits in these tasks. Crucially, our diagnostic analysis disentangles these failure modes. It demonstrates that deficits stem from limited context comprehension rather than insufficient intrinsic generative capability. To bridge this gap, we propose a training-free attention intervention strategy. Ultimately, $\textbf{GENIUS}$ establishes a rigorous standard for $\textit{GFI}$, guiding the field beyond knowledge utilization toward dynamic, general-purpose reasoning. Our dataset and code will be released at: $\href{https://github.com/arctanxarc/GENIUS}{https://github.com/arctanxarc/GENIUS}$.

1. Introduction

GENIUS addresses the lack of formal definitions, adequate benchmarks, and systematic analyses for evaluating Generative Fluid Intelligence in unified multimodal models. It introduces a three-dimensional, five-task benchmark, evaluates 12 models, diagnoses attention-related failures, and proposes a training-free intervention.

Motivation: GENIUS targets an evaluation gap because UMM research primarily measures Crystallized Intelligence—memorization and retrieval—rather than novel fluid reasoning.Existing work lacks a formal definition, adequate benchmarks that disentangle static knowledge, and systematic failure-mode analyses.
Benchmark design: GENIUS operationalizes fluid intelligence through three dimensions, five tasks, and hybrid metrics measuring Rule Compliance, Visual Consistency, and Aesthetic Quality.The three dimensions derive from Inductive Inference, Abstract Dynamic Reasoning, and Adaptive Inhibition, while Figure 1 summarizes the benchmark hierarchy.
Empirical findings: Evaluation of 12 open-source and proprietary models reveals clear gaps between current SOTA systems and general intelligence, with pre-planning and post-reflection yielding marginal gains.Manually annotated hints underwent at least three rounds of cross-validation to support unbiased hybrid evaluation.
Failure analysis and intervention: Attention visualizations suggest irregular noise and spikes across multimodal context, motivating a training-free mechanism that consistently improves performance across all tasks.The proposed explanation links imbalanced attention to insufficient guidance during implicit gradient descent and difficulty overcoming pretrained priors.
Contributions: GENIUS fills key gaps by formally defining GFI and introducing the first benchmark dedicated to its systematic, pure evaluation.The benchmark contains 510 expert-curated samples spanning three dimensions and five tasks, with a hybrid evaluation protocol.

2. GENIUS

GENIUS defines Generative Fluid Intelligence as visual generation that induces implicit patterns, executes ad-hoc constraints, and adapts to contextual knowledge, then evaluates these abilities without relying on prior knowledge. Its 510 expert-curated samples span 20 sub-tasks and use multimodal contexts plus rule-compliance, visual-consistency, and aesthetic-quality metrics.

GENIUS: GENIUS is the first benchmark dedicated to evaluating Generative Fluid Intelligence in unified multimodal models.It targets limitations in inducing patterns, reasoning under novel constraints, and adapting to contextual information.
GENIUS: The benchmark contains 510 expert-curated samples across 20 sub-tasks: 86 for implicit pattern induction, 213 for ad-hoc constraint execution, and 211 for contextual knowledge adaptation.GENIUS is designed to exclude prior knowledge rather than emphasize static world knowledge, generation quality, or safety.
Test-case design: Each test case uses an interleaved multimodal context in which removing any single modality makes the task unsolvable.Implicit Pattern Generation requires combining stylistic images with textual preferences, while relying on either modality alone produces incorrect or underdefined outputs.
Core dimensions: Ad-hoc Constraint Execution tests visual and symbolic generation under newly defined meanings, while Contextual Knowledge Adaptation tests prior-conflicting rules and literal-versus-metaphorical interpretation.These tasks use context-specific rules that may be absent from or contradict pretrained knowledge.
Evaluation: GENIUS evaluates outputs with rule compliance, visual consistency, and aesthetic quality using a hybrid protocol that combines frontier-LMM judgments with manually curated evaluation hints.The metrics assess adherence to precise constraints, preservation of key visual elements, and physical coherence including anatomy, lighting, and AI artifacts.

3. Experiment

GENIUS evaluates 12 open-source and proprietary multimodal models and finds substantial deficits in Generative Fluid Intelligence, despite strong aesthetic generation. Diagnostic experiments attribute failures primarily to context-execution gaps and prior-driven reasoning, while validating the LMM-based evaluator against human judgments.

Main Results: Nano Banana Pro scores 57.19 overall while Bagel scores 26.74, revealing substantial GFI deficits even in leading proprietary and open-source models.The benchmark evaluates Rule Compliance, Visual Consistency, and Aesthetic Quality using a weighted Overall score with RC:VC:AQ = 6:3.5:0.5.
Main Results: Models perform worst on Contextual Knowledge Adaptation, often defaulting to pre-trained priors when instructions contradict world knowledge or remap semantics.This “cognitive inertia” exposes difficulty arbitrating between contextual instructions and learned knowledge.
Main Results: High Aesthetic Quality alongside substantially lower Rule Compliance reveals an illusion of competence: visually appealing outputs can remain logically incorrect.GENIUS therefore emphasizes context comprehension and logically correct visual synthesis beyond beautiful pixel generation.
Diagnostic Analysis: Human-curated text and multimodal hints substantially improve performance, indicating that context comprehension is central to solving GFI tasks.Pre-planning and post-reflection provide only marginal gains, whereas explicit progressive guidance addresses the models’ failure more effectively.
Diagnostic Analysis: VQA probes indicate that generative failure primarily reflects an execution gap rather than insufficient comprehension, while LMM scores strongly align with human ratings.Pearson correlations are 0.9630 for NanoBanana Pro and 0.9659 for Bagel; Qwen2.5-VL-72B preserves relative rankings despite assigning lower absolute scores.

4. A Potential Solution

GENIUS deficits arise from disordered attention over multimodal context, which produces stochastic implicit gradient directions that fail to overcome pretrained priors. A training-free attention intervention rectifies this landscape and improves Bagel performance across nearly all dimensions, including a 6.18% Overall-score gain.

Existing models distribute attention irregularly, with stochastic spikes and noise across multimodal context, hindering precise capture of pivotal ad-hoc rules.Attention visualizations use image tokens generated during the process as queries over the entire context.
The method is guided by a Bagel-specific derivation connecting context processing, attention differences, and iterative gradient-descent updates.The derivation formalizes in-context learning as implicit fine-tuning within Bagel’s Mixture-of-Transformer architecture.
The analysis hypothesizes that imbalanced attention injects erroneous gradient components, making implicit descent stochastic and unable to overcome pretrained priors.The theoretical account treats in-context learning in multimodal generation as implicit updates to specific model parameters, with attention magnitude determining gradient-update norms.
The training-free solution uses Keyword Distillation, Relevance Mapping, and Bias Injection to emphasize task-critical visual tokens while suppressing noise.Bagel distills region-specific keywords, maps their semantic relevance to visual context tokens, and reweights attention accordingly.
6.18% Overall-score improvement validates that the proposed attention intervention consistently outperforms baseline Bagel across nearly all dimensions.The intervention sharpens attention toward critical tokens and deterministically steers implicit gradient updates.

5. Conclusion

GENIUS is introduced as the first benchmark dedicated to systematically quantifying Generative Fluid Intelligence, formalizing it into three core dimensions. The paper further attributes some failures to attention defects and proposes a training-free adjustment strategy to activate latent GFI capabilities.

Conclusion: GENIUS is the first benchmark dedicated to systematically quantifying Generative Fluid Intelligence through three dimensions: Implicit Pattern Induction, Ad-hoc Constraint Execution, and Contextual Knowledge Adaptation.The framework is grounded in Cattell-Horn-Carroll theory and is presented as a rigorous standard for assessing model capability in novel scenarios.
Conclusion: The paper partially traces model failures to attention mechanism defects during inference and proposes a training-free adjustment strategy that effectively activates latent GFI capabilities.
Conclusion: GENIUS is intended to serve as a pivotal testbed for future research, guiding models from crystallized memorization toward true general intelligence.

Impact Statement

The paper aims to advance evaluation of Fluid Intelligence in generative models by distinguishing Generative Fluid Intelligence from standard crystallized knowledge retrieval. It seeks to redirect the field toward adaptable, logic-grounded, and more robust AI systems while highlighting the “illusion of competence.”

GENIUS advances evaluation of Fluid Intelligence in generative models through a dedicated benchmark and theoretical framework.
The framework distinguishes Generative Fluid Intelligence from standard crystallized knowledge retrieval to promote adaptability and logic-grounded control.
The work supports more robust AI systems by highlighting the “illusion of competence.”

A. Benchmark Details … B. Detailed Qualitative Examples and Model Outputs

GENIUS combines a benchmark spanning three dimensions, five tasks, and 20 sub-tasks with systematic multimodal evaluation and qualitative comparisons across representative models. Its evaluation measures rule compliance, visual consistency, and aesthetic quality while screening for reference-image copying.

A.1. Data Statistics: GENIUS comprises 3 dimensions, 5 tasks, and 20 sub-tasks.
A.2. Evaluation Prompt: The evaluation framework uses Large Multimodal Models to assess three dimensions of generative quality.
A.2. Evaluation Prompt: Rule Compliance audits textual-visual alignment, verifying nouns, adjectives, and spatial constraints against modification requests.The audit is designed to ensure 100% compliance, with prompt details provided in Fig. 8.
A.2. Evaluation Prompt: Visual Consistency is evaluated according to the number of reference objects requiring preservation, after antiplagiarism screening for copied images.The screening begins with a pixel-level identity check before the Visual Consistency audit.
A.2. Evaluation Prompt: Aesthetic Quality assesses visual logic, rendering clarity, and realism, rewarding commercial-grade outputs and penalizing structural collapses or AI hallucinations.The corresponding prompt template is provided in Fig. 10.
B. Detailed Qualitative Examples and Model Outputs: The benchmark provides comprehensive qualitative examples for every sub-task, pairing complete inputs with evaluation hints and generated outputs.Each instance includes context and instruction, Rule Compliance and Visual Consistency hints, and outputs from six representative models, including Nano Banana Pro and Nano Banana.

C. Evaluation using Qwen2.5-VL-72B as Judge

Using Qwen2.5-VL-72B as judge lowers Overall Scores for every tested model, but preserves largely consistent relative trends and rankings, supporting GENIUS’s evaluation reliability.

C. Evaluation using Qwen2.5-VL-72B as Judge: The evaluation uses Qwen2.5-VL-72B (Bai et al., 2025) as an alternative judge to assess the GENIUS benchmark.
C. Evaluation using Qwen2.5-VL-72B as Judge: The Overall score aggregates tasks with the metric ratio RC:VC:AQ = 6:3.5:0.5.
C. Evaluation using Qwen2.5-VL-72B as Judge: Qwen2.5-VL-72B judging universally decreases Overall Scores, while model rankings and relative performance trends remain largely consistent.This supports the robustness and reliability of the GENIUS benchmark despite shifts in absolute scores.

D. Additional Experiments and Analysis … E. Related Work

Additional experiments show that input structure and contextual information substantially affect GENIUS performance, while related work situates the benchmark within fluid-intelligence theory, unified multimodal modeling, and limitations of existing generative evaluations.

D.1. Ablation on Interleaved Format: The benchmark defines Edit Mode, Interleaved, and Fine-Grained paradigms to examine how multimodal input structure affects model compatibility and performance.Edit Mode separates images from text, whereas the interleaved paradigms integrate modalities more closely.
D.1. Ablation on Interleaved Format: Performance varies across input formats and models, with a significant gap between Edit Mode and interleaved modes, while the two interleaved formats differ less.The comparison evaluates Nano Banana models and Bagel across the three formats defined in Fig. 6.
D.2. Discussion on the Composition of Input: Context removal causes a precipitous Rule Compliance decline across tasks, especially those requiring inductive reasoning or context-defined visual-textual mappings.Prior-Conflicting Generation also drops significantly because the model reverts to pre-trained priors without contextual definitions.
E. Related Work: Fluid Intelligence distinguishes novel-problem solving through inductive and dynamic reasoning from Crystallized Intelligence’s use of accumulated knowledge.This distinction provides the theoretical framing for evaluating generative fluid intelligence.
E. Related Work: Unified multimodal models have shifted from modular combinations of language models and diffusion decoders toward native multimodal fusion, including Chameleon (Team, 2024) and Show-o (Xie et al., 2024; Guo*).Earlier approaches bridged pretrained large language models with diffusion decoders for visual synthesis (Qin et al., 2024; Esser et al., 2024; Li et al., 2025b; Koh et al., 2023).
E. Related Work: Existing generative benchmarks assess UMM capabilities but largely remain confined to traditional evaluation paradigms (Ghosh et al., 2023; ZhaoThe cited benchmark landscape motivates evaluating capabilities beyond conventional single-image generation and basic text-to-image alignment.
E. Related Work: Early benchmarks such as GenEval (Ghosh et al., 2023), WISE (Niu et al., 2025), and DPG-Bench (Hu et al., 2024) focus primarily on single-image generation, static world knowledge, or basic text-to-image alignment.These evaluations do not involve complex, interleaved contexts.

F. Details of Method · F.1. Prompt Template for Keyword Generation · F.2. Mathematical Formulation of Attention Modulation

The method extracts task-critical visual cues as keywords, converts them into relevance scores, and uses these scores to modulate attention selectively during generation. The modulation injects a dynamic, bipolar bias into selected decoder layers and steps, with Softmax suppressing attention to noise tokens.

F.1. Prompt Template for Keyword Generation: The method uses a prompt to guide Bagel in identifying key regions in context images, whose generated keywords support subsequent relevance-map computation.These keywords provide the task-critical visual cues used by the attention-modulation stage.
F.2. Mathematical Formulation of Attention Modulation: The Bias Injection stage adapts a modulation strategy mathematically inspired by Li et al. (2025e;c) to keyword-based relevance scoring.
F.2. Mathematical Formulation of Attention Modulation: Attention modulation is applied selectively to decoder layers Lselected and generation steps Tselected, targeting head h in layer l ∈ Lselected at step t ∈ Tselected.The original attention logits Al,h ∈ R^N×N and relevance score vector S ∈ R^N are defined for each targeted head and step.
F.2. Mathematical Formulation of Attention Modulation: A dynamic bias term is injected into the attention mechanism to produce modulated attention logits ˆAl,h and enforce focus on critical signals.
F.2. Mathematical Formulation of Attention Modulation: The modulation uses query index i, key index j, and scalar λ to control intensity, while F(·) maps raw relevance scores to a bipolar bias distribution.
F.2. Mathematical Formulation of Attention Modulation: The relevance scores are normalized using their context-window mean µS and standard deviation σS, after which final attention weights are computed with Softmax.
F.2. Mathematical Formulation of Attention Modulation: The formulation dampens noise-token gradient contributions through their exponential suppression by the Softmax function.

G. Theorem Part … Æ Prompt Template for Keyword Generation

The appendix defines Bagel’s intermediate attention computation, proves two update-rule theorems through algebraic expansions, and specifies a prompt template for extracting image roles and keywords. It also documents evaluation prompt templates and qualitative examples.

G.1. Exact Definition of A: Bagel’s MoE formulation defines noisy intermediates from text context and image inputs, with separate Understanding and Generation query, key, and value projections.The attention dimension is denoted d_attn, and expert-specific projections operate on U and G components.
G.1. Exact Definition of A: The decoder uses Pre-Layer Normalization and concatenates intermediate representations before applying its detailed layer update rule.The formulation sets the initial bias to b_initial = 0 and uses RMS normalization and an Up layer.
G.1. Exact Definition of A: The core attention function A(u, g) is explicitly formulated from the query, key, and value components of U and G.These components correspond to expert-specific projections within the MoE architecture.
G.2. Proof of Thm. 4.1: The proof of Theorem 4.1 expands the perturbed decoder update into a form involving A(u, g), normalized attention, Up(δA), and bias terms.Equation (25) is expanded to Equation (27), after which the theorem proof is completed.
G.3. Proof of Thm. 4.2: The proof of Theorem 4.2 separately expands the Up update and bias update, applies a stated property, and concludes the theorem.The derivation proceeds from the direct Up expansion through defined quantities and the corresponding property for b.
Æ Prompt Template for Keyword Generation: The keyword-generation prompt instructs an image-generation planner to parse multimodal instructions and assign each provided image a specific retained role.It distinguishes a full target canvas or base image, represented by “all,” from extraction of specific features such as a face, shirt, or cup.
Æ Prompt Template for Keyword Generation: The appendix also provides prompt templates for rule compliance, visual consistency, and aesthetic quality evaluation, alongside detailed qualitative model-output examples.These materials are presented in Figures 8–12.