UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?

• Existing benchmarks often reduce visual inputs to language and evaluate generation and understanding separately, missing reasoning that requires non-linguistic visual transformations.UniG2U targets cases such as geometric auxiliary constructions, spatial-layout reconstruction, and intermediate states in puzzles and games.Source: §1 · p. 1
• UniG2U evaluates Generation-to-Understanding with matched unified-model and base-VLM protocols across seven reasoning regimes and over 30 models.The testbed contains 3,000 curated instances and supports reproducible evaluation with standardized data, protocols, and scripts.Source: §1 · p. 2
• Unified models generally underperform their base VLMs, while Generate-then-Answer typically performs worse than direct inference and can propagate visual errors.These findings expose trade-offs between generative and discriminative objectives and vulnerabilities in GtA visual reasoning.Source: §1 · p. 2
• Generation consistently helps on selected spatial, illusion-sensitive, and multi-round reasoning subtasks requiring visual transformations.Enhanced spatial and shape perception and multi-step intermediate image states provide the main benefits in these settings.Source: §1 · p. 2
• G2U gains cluster by cognitive demand, with perception tasks correlating with perception tasks and logic- or reasoning-intensive tasks correlating with one another.The results indicate that generation-understanding coupling induces class-consistent inductive biases shaped primarily by pretraining data and secondarily by architecture.Source: §1 · p. 2

2.1 Unified Multimodal Models

• Unified multimodal models evolved from disjoint vision–language and generation pipelines toward shared-interface architectures supporting both understanding and synthesis.Token-based early fusion and hybrid autoregressive–diffusion or flow formulations broaden modality coverage, while surveys identify evaluation as a key bottleneck.Source: §2.1 · pp. 2–3

2.2 Benchmarks for Unified Models

• Existing benchmarks evaluate unified capabilities through coverage, mixed outputs, bidirectional synergy, or reciprocal reasoning, but none systematically diagnoses G2U across fine-grained regimes and model archetypes.They differ in scope and protocol: some lack synergy isolation, others cover narrower task types, use limited scale, omit base-VLM pairing, or target mixed-output quality rather than causal G2U effects.Source: §2.2 · p. 3
• UniG2U isolates G2U by pairing unified models with purely discriminative base VLMs and curating regimes such as spatial manipulation, state externalization, and visual illusions.This framework reveals class-consistent inductive biases and identifies scenarios where generation supports multimodal reasoning.Source: §2.2 · p. 3
• UniG2U is the most comprehensive benchmark to date for evaluating unified models, featuring 3,000 understanding tasks, 30 fine-grained subtasks across 7 reasoning regimes, and over 30 models.Source: §2.2 · p. 3

3.1 Definitions: UMMs and G2U

• Each task instance is represented as x = (V, T, Q), with visual context V, text context T, query Q, and target answer y ∈ Y.Source: §3.1 · p. 3
• UMMs combine multimodal understanding with a generative interface that produces intermediate visual artifacts, which may shape representations or support controllable reasoning.Artifacts can include images, sketches, annotations, reconstructions, or edits.Source: §3.1 · p. 3
• G2U gain is defined as ΔG2U(MUM; D) = Perf(MUM; D) − Perf(B(MUM); D), comparing each unified model with its matched purely discriminative base VLM.Matching capability and computation budgets is intended to isolate generation-to-understanding effects from model scale, architecture, pretraining, or empirical noise.Source: §3.1 · p. 4

3.2 Taxonomy of Unified Multimodal Models

• The taxonomy distinguishes E2E models, Decoupled systems, and agentic unified models according to how generation and answering are implemented and trained.E2E systems jointly train shared parameters; Decoupled systems stitch an explicit generation module to a frozen answering module; UM-Ag systems coordinate separate models through protocols or tools.Source: §3.2 · pp. 4–5
• Agentic systems are separated from standard unified models because their external tool use and distinct, often closed-source models prevent a strictly matched base-VLM comparison and exact G2U-gain calculation.Source: §3.2 · p. 5

3.3 Direct vs. Generate-then-Answer (GtA) Inference

• Direct inference answers from x without an external artifact, whereas Generate-then-Answer explicitly generates and re-consumes an intermediate visual rationale before answering.Agentic GtA can instead use a closed-loop rollout with multiple generated artifacts.Source: §3.3 · p. 5
• The overall G2U shift separates ΔDirect, the intrinsic effect of unified training, from ΔGtA, the effect of explicitly externalizing visual reasoning before answering.GtA may hurt performance when generated artifacts are inaccurate, physically implausible, or misaligned with the intended reasoning plan, misleading the answering module.Source: §3.3 · p. 5

4.1 Dataset Construction and Task Taxonomy

• UniG2U contains approximately 3,000 samples organized into 7 high-level categories and 30 fine-grained subtasks for diagnosing generation-to-understanding across perceptual and cognitive structures.The taxonomy includes generation-friendly and perception-heavy regimes, enabling analysis of when intermediate visual generation is helpful, neutral, or harmful.Source: §4.1 · p. 5, §4.1.1 · p. 6

4.1.1 Design Principles

• The taxonomy separates distinct cognitive structures and selects tasks where visual externalization is intrinsically coupled with comprehension, including geometric construction, physical simulation, spatial transformation, and perceptual reconstruction.This design follows the hypothesis that some understanding tasks benefit from intermediate visual generation rather than aggregating arbitrary multimodal tasks.Source: §4.1.1 · p. 5, §4.1.2 · p. 6

4.1.2 Task Selection and Coverage

• The benchmark balances seven categories from 100 Chart & Table samples to 1,263 Perception Reasoning samples, deliberately filtering scarce instances with genuine G2U synergy.Category counts are 200 each for Real-world Applications, Geometry Reasoning, and Physics Reasoning; 537 for Puzzles and Games; and 500 for Spatial Intelligence.Source: §4.1.2 · p. 7

4.2 Evaluation Protocol

• The evaluation compares Direct inference with Generate-then-Answer (GtA) under matched budgets, using iterative intermediate states for inherently multi-step tasks and standardized decoding where possible.Direct produces the final answer in one pass, whereas GtA generates a visual artifact before answering with the original input and artifact as context.Source: §4.2 · p. 8

4.3 Metrics

• Accuracy is paired with G2U Gain, while RA and AL score intermediate-image fidelity and consistency with the final answer and query on 1-to-5 rubrics.G2U Gain measures a unified model’s accuracy improvement relative to its rigorously paired base VLM, overall and per subtask.Source: §4.3 · pp. 8–9

4.4 Models and Implementation Details

• The study evaluates 35 models—11 base VLMs, 21 native unified models, and 3 agentic models—using corresponding or closest available VLMs as paired baselines.Agentic systems provide a generalized unified-model ceiling and test tool-driven visual externalization, while all systems share the dataset, prompts, and budget constraints.Source: §4.4 · pp. 9–10

4.5 Research Questions

• Four research questions test unified-versus-base performance, GtA-versus-Direct gains, architecture-linked behavioral correlations, and whether intermediate visualizations are faithful and useful.The study uses matched inference for the first two questions, rank correlations across architectural classes, and RA/AL correlations with accuracy and GtA gains.Source: §4.5 · p. 10

5.1 Main Results (RQ1)

• Unified models generally underperform paired base VLMs, with negative Δ persisting under Direct inference and indicating an intrinsic alignment tax from generation–understanding coupling.The degradation is attributed to objective interference and parameter-level coupling rather than catastrophic forgetting or low-quality generated images.Source: §5.1 · pp. 11, 13, Table 4 · p. 12
• Despite overall decline, unified models show resilience or gains in Spatial Intelligence and Visual Illusions, where generation regularizes geometric transformation, state tracking, and visual-structure modeling.The benefit depends on coherent alignment between generative and understanding objectives; otherwise, generation dilutes general reasoning.Source: §5.1 · p. 13, Table 5 · p. 13

Comparison Between GtA and Direct Inference (RQ2)

• GtA degrades most logic-intensive tasks because inaccurate or misaligned generated artifacts propagate visual errors, whereas it improves Maze, Sliding puzzles, and Multi-Step Spatial Reasoning through visual chain-of-thought.Direct inference avoids committing to flawed images, while GtA helps when intermediate states provide a reliable external workspace for sequential spatial manipulation and state tracking.Source: §5.2 · pp. 13–14

5.3 Correlation Analysis of G2U Gains Across Tasks and Models (RQ3)

• G2U gains form internally correlated perception and reasoning clusters that are typically negatively correlated, while models sharing the same base VLM correlate far more strongly than models sharing only a generative architecture.These patterns suggest inherited foundational representations govern generation-induced effects more strongly than the chosen generative paradigm.Source: §5.3 · pp. 14–15, Figure 10 · p. 15

5.4 Are Intermediate Visualizations Aligned and Helpful? (RQ4)

• High visual alignment is necessary but insufficient: perception tasks produce faithful yet redundant drawings, whereas low-alignment structural tasks suffer error propagation; gains arise when visual scaffolds are both needed and trustworthy.Spatial Intelligence occupies this beneficial sweet spot, with visual externalization reducing cognitive load without introducing unreliable intermediate premises.Source: Table 7 · p. 14, §5.4 · pp. 15–16

• UniG2U evaluates generation-to-understanding across 3,000 curated instances spanning seven cognitive regimes.It is designed as a large-scale diagnostic benchmark for studying when and how generation helps understanding in unified multimodal models.Source: §6 · p. 16
• Unified models do not consistently outperform base VLMs; gains arise mainly in spatial transformation and state-tracking tasks.Generation-trained representations may regularize structural reasoning, but the benefits remain task- and architecture-dependent.Source: §6 · p. 16
• Generate-then-Answer does not universally surpass direct inference because inaccurate or misaligned visual artifacts can introduce failure modes.Intermediate visual externalization can assist reasoning, but it is not reliably beneficial.Source: §6 · p. 16
• Future work should strengthen generation–reasoning integration through alignment, self-verification, closed-loop refinement, multimodal artifacts, scaling-law analysis, and causal evaluation.These directions aim to mitigate error propagation, improve robustness, and isolate representation changes.Source: §6 · p. 16

Performance comparison

• Frozen-backbone unified models show measurable deviations from the expected qwen2.5-vl-7b base-model performance in Direct (Non-CoT) evaluation.The expected match follows from bypassing the generative pathway while keeping the base VLM backbone frozen.Source: Table 8 · p. 19, A Performance Analysis of Unified… · p. 19

Implementation factors

• EMA checkpoints or other subtle weight differences can produce slight performance drift from the original base model.Using weights that are not exactly those of the original base model can alter predictions even when the backbone is nominally frozen.Source: A Performance Analysis of Unified… · p. 19
• Inference hyperparameters, software versions, image preprocessing, and numerical precision can shift outputs and accumulate errors, including through bfloat16 RoPE computation.Relevant factors include generation length, image scaling, decoding, PyTorch/CUDA/transformers versions, image encoding, and precision-dependent logit changes.Source: A Performance Analysis of Unified… · p. 19

Ablation findings

• 2.33% and 10.99% overall drops occur for UAE and X-Omni under OOD-Image generation, while TokenFlow-XL drops 4.65% with Prompt-Only generation versus Direct inference.These results are summarized in Table 9 and cover forcing intermediate image generation for UAE and X-Omni, versus text-only conditioning for TokenFlow-XL.Source: Table 9 · p. 19, B Ablation Study for Models without… · p. 20
• Figure 11 shows that edit-untrained intermediate images are often chaotic, structurally irrelevant, or information-deficient, misleading rather than assisting subsequent reasoning.These qualitative failures corroborate the quantitative declines reported in Table 9.Source: Figure 11 · p. 19, B Ablation Study for Models without… · p. 20

Evaluation rationale

• Edit-untrained models are evaluated separately because image-editing intermediate generation falls outside their training distribution and creates an OOD setting.The authors therefore avoid placing these Generate-then-Answer scores directly in the main comparison, where they could introduce unfair bias.Source: B Ablation Study for Models without… · p. 20

Ablation design

• OOD-Image generation forces UAE and X-Omni to generate from the original image, whereas Prompt-Only generation makes TokenFlow-XL generate solely from the textual prompt before answering.Both generated images are then re-consumed as additional context for the answering stage.Source: B Ablation Study for Models without… · p. 20

Implication

• The ablations validate excluding edit-untrained Generate-then-Answer scores and show that G2U gains critically depend on explicit, training-aligned intermediate generation.Without such capabilities, intermediate generation can degrade downstream reasoning instead of improving it.Source: B Ablation Study for Models without… · p. 20

• G2U gains are most consistent in Spatial Intelligence and illusion-related perceptual reasoning, although overall improvements are not universal.These subtasks involve transformation-aware reasoning, geometric consistency, or resolving visually ambiguous patterns.Source: C Additional Result Analysis:… · p. 20
• Generation-induced representation coupling may improve robustness to visually ambiguous or structurally complex inputs.The proposed mechanism is enhanced internal spatial regularization from generation-trained representations.Source: C Additional Result Analysis:… · p. 20
• Several unified variants outperform their corresponding base models on spatial perception, while logo- and shape-sensitive illusion subtasks improve in selected configurations.Examples include UniWorld-V1, JavisGPT, uni-video, and STAR-7B in the spatial Perception Reasoning subtask.Source: C Additional Result Analysis:… · p. 20
• Recognition-heavy subtasks such as icon_scene and in_scene show no systematic improvement and sometimes mildly degrade.Thus, G2U benefits emerge primarily in transformation-centric or structurally constrained regimes.Source: C Additional Result Analysis:… · p. 20

Taxonomy

• The taxonomy separates failures into capability errors, semantically relevant but non-useful outputs, and irrelevant generation according to validity, utility, and relevance.Successful generation must be structurally valid, semantically relevant, and useful for reasoning; the three categories violate these criteria in order.Source: D Failure Case Taxonomy for… · pp. 20, 22

Category I: Capability Failure

• Capability failure produces geometrically distorted, quantitatively inconsistent, physically invalid, or constraint-missing diagrams that can corrupt downstream computations.Solvers may treat invalid diagrams as ground truth, leading to incorrect intermediate reasoning or forcing them to ignore the intended externalization benefit.Source: D Failure Case Taxonomy for… · p. 22
• Capability failures expose a diagram-grammar bottleneck, suggesting validation or self-checking before solving and fallback to pure-text reasoning when validity is uncertain.Useful checks include right angles, axis monotonicity, and connectivity constraints.Source: D Failure Case Taxonomy for… · p. 22

Category II: Surface-Relevant but Non-Utility Generation

• Surface-relevant but non-utility generation produces on-topic diagrams that restate the prompt, omit operational structure, or add aesthetic detail without making the required computation easier.Relevance alone does not guarantee utility: such outputs can waste generation budget or distract reasoning with non-actionable details.Source: D Failure Case Taxonomy for… · p. 22
• Category II calls for goal-conditioned generation of minimal task-specific structure, with budget-aware routing that skips generation when predicted utility is low.Examples include annotated axes, explicit path graphs, or marked equal angles rather than generic illustrations.Source: D Failure Case Taxonomy for… · p. 23

Category III: Irrelevant Generation

• Irrelevant generation reflects complete task misalignment, including drawing for purely textual queries or producing images that lack required chart schemas and introduce spurious content.It can waste compute, divert attention, reduce trust, and complicate debugging; modality-aware routing should avoid image generation for pure-text tasks.Source: D Failure Case Taxonomy for… · pp. 23–24

• Uni-MMMU comprises Jigsaw, Maze, and Sliding Puzzle tasks, each assessing different aspects of visual reasoning through interleaved image-text generation.Source: E Uni-MMMU Task Design · p. 24

E.1.1 Task Description

• Jigsaw completion selects which candidate correctly completes the reference image, evaluated with binary jigsaw_text_acc accuracy on 100 samples.The required output is JSON containing choice 0 or 1 and a rationale explaining the decision.Source: §E.1.1 · p. 24

E.1.2 Visual CoT Generation Flow

• The jigsaw visual CoT flow generates two completed 2×2 images, one per candidate, while preserving the other three reference cells, then compares them to produce the final answer.The illustrated rationale identifies better seam continuity as the decision basis.Source: §E.1.2 · p. 24

E.2.1 Task Description

• Maze solving finds legal up, down, left, or right paths from a blue start dot to a green goal through white walkable cells without crossing black walls, evaluated by maze_text_exact and maze_text_frame_acc on 100 samples.The output is an ordered move sequence with one-cell, non-diagonal transitions.Source: §E.2.1 · p. 25

E.2.2 Visual CoT Generation Flow

• The maze visual CoT flow generates one image after each move, changing only the blue-dot position and excluding the initial state, collages, arrows, and annotations.The number of generated images must equal the number of moves, with each image representing exactly one step.Source: §E.2.2 · p. 25

E.3.1 Task Description

• The sliding-puzzle task solves a 3×3 configuration with eight colored tiles and one empty red space, using tile-movement directions and reporting sliding_text_exact and sliding_text_frame_acc over 54 samples.Moves are named by the colored tile’s direction, not by the empty space, and must be legal single-tile steps.Source: §E.3.1 · p. 26

E.3.2 Visual CoT Generation Flow

• The sliding-puzzle visual CoT flow generates one image per move, changing only tile positions while preserving colors pixel-identically, then emits the move sequence without an initial state or multi-panel images.The supplied tables are labeled as main and delta results across UniG2U categories, but their numerical cells are not provided.Source: §E.3.2 · p. 27

G.1.1 mmsi_attribute_appr (Attribute - Appearance) and mmsi_attribute_meas (Attribute - Measurable) and mmsi_msr (Multi-Step Reasoning)

• The direct experimental prompts emphasize careful visual observation and output only the selected option letter, while motion prompts specifically analyze consecutive first-person images to determine camera movement or rotation.The direct formats cover appearance, measurable attributes, multi-step reasoning, and camera motion; the camera and object-motion tasks share the same non-CoT prompt structure.Source: §G.1.1 · p. 32, §G.1.2 · p. 32

G.2.1 ICON Shape Recognition

• Recognition prompts ask models to identify an icon, scene, logo, or shape from integrated foreground-background images using constrained single-word class choices.Shape and scene recognition prompts define candidate class lists and require the single most accurate class name; the Visual CoT version additionally annotates how background elements form the icon silhouette.Source: §G.2.1 · p. 33, §G.2.2 · p. 33, §G.2.3 · p. 34

G.4.1 uni_mmmu_jigsaw100_visual_cot

• Jigsaw, maze, and sliding-puzzle Visual CoT tasks require explicit image-state generation: candidate compositing for jigsaws, one image per legal maze move, and one image per sliding-tile move followed by a strict JSON move answer.The maze and sliding-puzzle prompts preserve the input layout or tile colors while changing only the relevant state, forbid collages, and require exact move-list formatting.Source: §G.4.1 · p. 35, §G.4.2 · p. 36

H Task Prompts (Visual CoT Version)

• Visual CoT tasks use a two-stage format in which a generation prompt produces an intermediate visualization and a question prompt uses it with the original image before answering.The appendix defines the two-stage [GEN_PROMPT]...[/GEN_PROMPT][QUESTION]...[/QUESTION] structure for Visual CoT tasks.Source: H Task Prompts (Visual CoT Version) · p. 37

H.1.1 mmsi_attribute_appr_visual_cot

• Appearance, measurable-attribute, and multi-step-reasoning tasks direct models to inspect visual attributes, quantitative relationships, or annotated intermediate reasoning steps before selecting an answer option.The corresponding visualizations emphasize color, shape, texture, orientation, counts; size, length, width, height, distance, area, and volume; or spatial relationships and changes across images.Source: §H.1.1 · p. 37, §H.1.2 · p. 37, §H.1.5 · p. 38

H.1.3 mmsi_motion_cam_visual_cot

• Motion Visual CoT tasks generate camera-motion or object-motion visualizations, then use them to infer viewpoint changes, object directions, relative speeds, and trajectories before answering.Camera prompts request arrows and shifted reference points, while object prompts request motion trails, directional arrows, bounding boxes, speeds, and trajectories.Source: §H.1.3 · pp. 37–38, §H.1.4 · p. 38

H.2.1 ICON Shape Recognition Visual CoT

• Recognition tasks generate visualizations of icon, scene, logo, or object-shape structure, then use the original and visualization to classify the target.The prompts cover ICON, LOGO, and IN shape and scene recognition, with constrained class-name responses.Source: §H.2.1 · p. 39, §H.2.2 · p. 39, §H.2.3 · p. 39

H.3.1 Algorithmic Visual CoT

• Reasoning tasks generate diagrams exposing algorithmic sequences, analogical transformations, deductive inference chains, inductive patterns, or spatial transformations before selecting an answer option.Auxiliary diagrams show rules, transformations, premises, progression, rotations, folding, viewpoints, and orientation markers, after which models answer directly with an option letter.Source: §H.3.1 · p. 40, §H.3.2 · p. 41, §H.3.3 · p. 41

H.4.1 chartqa100_visual_cot

• Chart and geometry procedures annotate relevant chart values or add auxiliary geometric constructions, then provide both original and enhanced diagrams for solving.Geometry prompts emphasize labeled lines, points, angles, segments, and properties such as parallelism, congruence, and equal angles.Source: §H.4.1 · p. 43, §H.5.1 · p. 43

H.5.2 auxsolidmath_easy_visual_cot

• Solid-geometry procedures preserve the original 3D figure while adding colored or dashed auxiliary lines, then use the resulting relationships for step-by-step calculations and a final numerical answer.Suggested constructions include perpendiculars, connecting segments, midpoints, extensions, parallels, and cross-sections.Source: §H.5.2 · p. 44

H.6.1 phyx_optics100_visual_cot

• Physics procedures generate domain-specific auxiliary diagrams: light-ray paths for optics and force-analysis diagrams for mechanics, each supplied alongside the original problem image.The optics diagrams include incident, reflected, and refracted rays, focal points, and angles; mechanics diagrams show forces and their directions or relative magnitudes.Source: §H.6.1 · pp. 44–45, §H.6.2 · p. 45

H.7.1 realunify_mental_tracking_visual_cot

• Mental-transformation procedures either apply requested image modifications and present the transformed image or reconstruct an image shuffled by patches without adding or altering content.Both procedures use a generated image in addition to the original or restoration instruction.Source: §H.7.1 · p. 45, §H.7.2 · p. 45

H.7.2 realunify_mental_reconstruction_visual_cot; H.7.3 realunify_attentional_focusing_visual_cot

• Reconstruction and attentional-focusing tasks provide restored or relevance-highlighted visualizations alongside the original image before answering.These prompts target shuffled-image restoration and question-relevant region highlighting.Source: §H.7.2 · p. 45, §H.7.3 · p. 45

H.8.1 vsp_google_map_visual_cot; H.8.2 vsp_collision_visual_cot

• Navigation tasks generate auxiliary visual analyses for map routes or car-person distances, using directional moves, crossroads, grid distances, and speeds to answer path or timing questions.The map task requires an ordered direction-and-crossroad output, while collision analysis computes each agent’s travel time separately.Source: §H.8.1 · p. 46, §H.8.2 · p. 47

H.9.1 babyvision_fine_grained_visual_cot; H.9.2 babyvision_visual_tracking_visual_cot

• BabyVision tasks generate visualizations that emphasize fine-grained discriminative details or track relevant objects, followed by step-by-step answers in boxed format.Both tasks provide the auxiliary visualization together with the original image.Source: §H.9.1 · p. 48, §H.9.2 · p. 48

H.10.1 uni_mmmu_jigsaw100_visual_cot; H.10.2 uni_mmmu_maze100_visual_cot; H.10.3 uni_mmmu_sliding54_visual_cot

• Interactive puzzle protocols require exact visual transformations and strict outputs: candidate insertion and comparison for jigsaws, one image per legal maze move, and one image per sliding-puzzle move.Jigsaw comparison assesses seam continuity, color/texture gradients, structural alignment, and global semantics, while maze and sliding tasks prohibit collages and require final JSON move lists.Source: §H.10.1 · pp. 48–49, §H.10.2 · p. 49, §H.10.3 · p. 50

I.1 Purpose; I.2 Per-sample Inputs and Metadata; I.3 Metric Definitions

• The appendix defines prompt-based multimodal-LLM judging with RA for generation-instruction usefulness and AL for answer consistency with visual evidence and question intent.Dataset-level averages are computed from per-sample metadata containing task prompts, generated-image paths, questions, and stage-two answers.Source: §I.1 · p. 50, §I.2 · p. 51, §I.3 · p. 51

I.4 End-to-end Evaluation Flow

• End-to-end evaluation loads metadata, resolves generated-image paths, and constructs a category-specific RA prompt by adding it to the base RA prompt.The documented flow is prompt-based and operates over the stored sample metadata.Source: §I.4 · p. 51
• The end-to-end flow constructs category-aware prompts, judges original and auxiliary images, parses reasoning and answer visual scores, and saves both scores with rationales.The saved schema includes ra_score, ra_reasoning, al_score, and al_reasoning.Source: §I.4 · p. 51, §I.10 · p. 56

I.5 Judge Call Settings

• The multimodal judge uses a GPT-4o-compatible Azure OpenAI endpoint with 3000 response tokens, JPEG-base64 images, and metric- and API-level retries with backoff.Source: §I.5 · p. 51

I.6 Prompt Composition Rule

• Prompt composition prioritizes explicit task-category criteria, then legacy task-name mappings, and finally the base prompt when no category information is available.Source: §I.6 · p. 51

I.7 Complete Base Prompt: RA

• The RA prompt evaluates whether generated visualization images follow the generation prompt through instruction following (40%), visual quality (30%), and relevance to the task (30%), not final-answer correctness.Source: §I.7 · p. 52

I.8 Complete Base Prompt: AL

• The AL prompt evaluates whether the final answer is consistent with generated visuals and the original question, prioritizing visual-answer consistency, alignment, and reasoning coherence.Visual-answer consistency receives 50%, question-answer alignment 30%, and reasoning coherence 20%.Source: §I.8 · p. 53

RA addition

• RA additions assess visualization-specific criteria across real-world, mathematical, physics, puzzle, chart, spatial, and perception reasoning domains.Criteria cover safety and feasibility, formula and scientific accuracy, rule adherence, data highlighting, spatial transformations, and perceptual clarity.Source: RA addition · pp. 54–55

AL addition

• AL additions assess answer validity against visuals and domain requirements, including practical safety, mathematical correctness, scientific evidence, puzzle rules, chart data, spatial transformations, and perceptual attributes.The criteria also require answers to avoid hallucinated data and remain consistent with shown evidence, geometry, and scenes.Source: AL addition · pp. 54–55

I.10 Scoring Outputs and Schema

• Outputs are stored as document, task, and JSON-path metadata alongside nullable 1–5 RA and AL scores and their textual reasoning fields.Source: §I.10 · p. 56

• If JSON parsing of a judge response fails, the pipeline falls back to regex extraction.Source: §I.11 · p. 56
• If all retries fail, the score is returned as null.Source: §I.11 · p. 56
• For multi-image tasks, all original images are concatenated before auxiliary images.Source: §I.11 · p. 56