PaperlayerView exact revision on arXiv

Source-linked AI summary

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Zefan Cai, Haoyi Qiu, Tianyi Ma, Haozhe Zhao, Gengze Zhou, Tingting Liao, Xinyan Velocity Yu, Kung-Hsiang Huang, Ke Wan, Shawn Lin, Parisa Kordjamshidi, Minjia Zhang, Wen Xiao, Jiuxiang Gu, Nanyun Peng, Junjie Hu

arXiv:2512.14691v3cs.CLcs.CV

TL;DR

It remains unclear whether fluent generated images and videos reflect reasoning that preserves task constraints. MMGR evaluates image, video, and language-based systems across 10 tasks with answer-verifiable and video process-aware scoring. Results show a gap between visual quality and reasoning correctness, including weak video performance on symbolic tasks and strong image-generator navigation scores.

  • Problem

    Existing evaluations are fragmented across modalities and often focus on final outputs, leaving uncertainty about whether generated media solves the reasoning task and whether video intermediate states are valid.

    Source: §1 · pp. 1–2

  • Method

    MMGR evaluates video and image generators across 10 tasks in three domains, using answer-verifiable scoring where possible and checking whether video frames form valid steps toward the target.

    Source: §1 · pp. 2–3

  • Results

    Video models perform best on Physical Commonsense but remain weak on symbolic tasks, while image generators lead final scores across all four navigation tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MMGR shifts evaluation from whether generated media looks convincing to whether it solves the reasoning problem it depicts.

    Source: §7 · p. 10

    Physical Commonsense and several embodied-navigation tasks rely solely on VLM evaluation, inheriting weaknesses in spatial reasoning and physical understanding.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

1 Introduction

Existing evaluations leave open whether generated media solves the underlying reasoning task and whether video’s intermediate states are valid. MMGR evaluates image and video generators across reasoning domains, using answer-verifiable scoring and process-aware checks for video.

1 Introduction

  • Existing benchmarks are fragmented across modalities, often score only outcomes, and struggle to distinguish task-solving from plausible-looking media or invalid intermediate video states.They also make it difficult to separate reasoning limits from modality-specific rendering constraints.Source: §1 · pp. 1–2
  • MMGR spans 10 tasks across Abstract Reasoning, Embodied Navigation, and Physical Commonsense, probing Physical, Logical, 2D Spatial, 3D Spatial, and Temporal reasoning.The suite compares video and image generators under a shared framework.Source: §1 · p. 2
  • MMGR uses answer-verifiable scoring where reliable ground truth exists and requires video intermediate frames to be valid steps toward the target, not merely smooth transitions.The framework is designed to compare video and image generators on the same task families.Source: §1 · p. 3
  • Video models perform best on Physical Commonsense but remain weak on symbolic tasks, while image generators lead final scores across all four navigation tasks.Sora-2 scores 76.00% on Physical Concepts and 64.00% on Sports; video models are also brittle under cross-view alignment and long-horizon spatial grounding.Source: §1 · p. 3

2 Related Work

Prior generative benchmarks span video and image tasks, but evaluations often focus on final outputs. MMGR connects process-aware scoring with cross-modal comparisons and real-world embodied navigation.

2 Related Work

  • Generative video benchmarks assess visual quality, alignment, motion, consistency, composition, or physical plausibility, while image benchmarks cover commonsense, spatial reasoning, editing, and generation-linked problem solving.Across modalities, evaluation often reduces to a final output scored by human, VLM, or checklist judgments, leaving intermediate steps unchecked.Source: §2 · pp. 3–4
  • MMGR spans Abstract Reasoning, Embodied Navigation, and Physical Commonsense, covering task families from mazes and math to navigation and sports.These domains provide a cross-domain scope alongside its cross-modal evaluation.Source: Figure 2 · p. 4, §2 · p. 5
  • Process-aware generative benchmarks examine intermediate frames or rule-checkable task steps, while visual-understanding and embodied-AI benchmarks ask models to interpret inputs or select actions rather than generate valid reasoning outputs.MMGR shares process-aware, rule-verifiable scoring with concurrent generative benchmarks but differs in scope.Source: §2 · p. 4
  • MMGR compares video, image, and LLM/VLM generators on the same tasks and adds real-world agent-centric navigation not covered by VIPER or VBVR.Its navigation settings include panoramic, top-down, and 3D real-world navigation, together with SLAG.Source: §2 · p. 5

3 MMGR Benchmark

MMGR groups its tasks into Abstract Reasoning, Embodied Navigation, and Physical Commonsense, aligning them with five reasoning abilities. The benchmark includes symbolic, navigation, and physical-interaction tasks.

3 MMGR Benchmark

  • MMGR organizes reasoning evaluation around symbolic rules and layouts, environment-consistent trajectories, and motion that respects intuitive object dynamics.Together, these domains exercise Physical, Logical, 2D Spatial, 3D Spatial, and Temporal reasoning.Source: §3 · p. 5

3.1 Domain 1: Abstract Reasoning

  • Abstract Reasoning comprises Maze, Sudoku, ARC, and Math tasks, with 240 mazes, 300 Sudoku puzzles, 456 ARC tasks, and 327 math problems.The tasks test path finding, constraint satisfaction, abstract rule induction, and mathematical reasoning, respectively.Source: §3.1 · p. 5

3.2 Domain 2: Embodied Navigation

  • Embodied Navigation covers panoramic last-mile, top-down, 3D real-world navigation, and SLAG, with 120 samples per task across 24 hard-level configurations.The configurations vary floor complexity, view fidelity, path length, and goal specification.Source: §3.2 · p. 5

3.3 Domain 3: Physical Commonsense

  • Physical Commonsense includes 25 VideoPhy physical-interaction examples and 25 compositional sports scenarios involving activities such as ballet, skiing, diving, and swimming.The sports scenarios involve balance, momentum, projectile motion, and fluid dynamics.Source: §3.3 · p. 6

4 Evaluation

MMGR evaluates zero-shot reasoning across 10 tasks using multiple outputs per prompt and a task-specific gated primary score. It combines deterministic checks with rubric-based VLM judging where exact verification is unavailable.

4 Evaluation

  • The evaluation benchmarks state-of-the-art generative models across 10 tasks to quantify performance and identify strengths and weaknesses across five reasoning dimensions.It is designed to assess zero-shot reasoning capabilities.Source: §4 · p. 6

4.1 Evaluated Models and Setup

  • The model pool includes video and image generators plus text-solution baselines on applicable abstract-reasoning tasks, with five outputs sampled per prompt and no task-specific tuning.Closed models use default settings and open models use recommended settings.Source: §4.1 · p. 6

4.2 Evaluation Protocol

  • Each task's primary score is gated: it equals 1 only when all required integrity, outcome, and reasoning-process metrics hold.Video evaluation covers final outcomes and, where relevant, intermediate-frame validity; image evaluation scores final outputs against task constraints.Source: §4.2 · p. 6
  • Tasks with exact ground truth use rule-based checks, while tasks without reliable pixel- or symbol-level verification use Gemini-2.5-Pro with task-specific rubrics.Maze paths are checked pixel-wise and Sudoku grids through OCR and constraint checks; the authors report human-evaluator agreement and reliability caveats for VLM judgments.Source: §4.2 · p. 7, VLM-Based Evaluation · p. 7

5 Evaluation Results

Generative models’ reasoning performance varies across domains: image generators lead navigation, video models are strongest on Physical Commonsense, and abstract reasoning remains difficult. Reliability is highest for deterministic checks, while VLM-based judgments make exact rankings less certain.

5.1 Automatic Evaluation

  • Scores are zero-shot under each task’s primary success metric, using deterministic checks for Maze and Sudoku and rubric-based VLM evaluation elsewhere.Text-only Gemini results are diagnostic baselines for abstract reasoning, not multimodal generation systems.Source: §5.1 · p. 7
  • Symbolic and constrained abstract tasks remain difficult: every video model scores 0.00% on Sudoku, while Nano-banana Pro reaches 72.69% on Math.Nano-banana Pro’s Math score is close to the Gemini text baselines of 71.38% and 74.14%.Source: §5.1 · p. 7
  • Image generators lead all four navigation settings, while video models drop sharply when viewpoints or horizons change.Veo-3 reaches 60.00% on Last-Mile but falls to 19.49% on Top-down View and 11.02% on SLAG.Source: §5.1 · p. 7
  • Video models perform best on Physical Commonsense: Sora-2 scores 76.00% on Physical Concepts and 64.00% on Sports, but VLM-based scores do not prove physical reasoning.The scores rely on VLM-based evaluation rather than a deterministic checker.Source: §5.1 · p. 7

5.2 Evaluation Reliability Analysis

  • Maze pixel agreement ranges from 95.56% to 100.00% and Sudoku OCR agreement is 100.00%, whereas navigation VLM agreement spans 21.67% to 88.45%, warranting caution on exact rankings.Physics reaches 65.31% agreement, so its scores are best read as judged plausibility rather than verified correctness.Source: Table 5 · p. 8, §5.2 · p. 8

6 Failure Cases and Future Directions

Failure cases expose a recurring gap between plausible media and valid task execution: models can break constraints, alter task states, or fail to maintain consistency across frames. The paper calls for generators and evaluations that directly address and verify task state.

6 Failure Cases and Future Directions

  • Across abstract reasoning and navigation, models often produce plausible media while violating task constraints or losing consistency across frames.Failures include altered inputs, broken global constraints, agent identity changes, inconsistent geometry, and goal-state drift.Source: §6 · p. 8
  • Veo-3 can reach navigation goals despite abnormal trajectories, floor-jumping physics violations, staircase alterations, or 2D trajectory misalignment.Figure 3 also shows a hallucinated second agent in Last-Mile navigation and Sudoku failures.Source: Figure 3 · p. 9
  • Future work calls for training signals and architectures that reward state preservation, constraint satisfaction, cross-view consistency, and completed trajectories, alongside metrics that verify task state.The paper also calls for longer, more controllable temporal horizons and training data beyond salient physical-commonsense clips.Source: §6 · p. 9

7 Conclusion

MMGR evaluates multimodal generation through answer-verifiable and process-aware tests of physical, logical, spatial, and temporal constraints. Its results reveal that compelling visuals do not ensure global consistency, symbolic abstraction, or long-horizon reasoning.

7 Conclusion

  • MMGR shifts evaluation from visual realism toward answer-verifiable, process-aware assessment across Abstract Reasoning, Embodied Navigation, and Physical Commonsense.For video, it uses chain-of-frame validity to test whether intermediate frames form valid steps toward an outcome.Source: §7 · p. 10
  • Models can produce visually compelling outputs yet fail tasks requiring global consistency, symbolic abstraction, and long-horizon reasoning.The conclusion characterizes these failures as reliance on perceptual shortcuts rather than grounded inference.Source: §7 · p. 10

Limitations

MMGR’s scope and evaluation design constrain what its results establish: it tests zero-shot text-to-image and text-to-video generation on structured tasks, with some outcomes judged only by VLMs. Its controlled, discrete tasks and sampled reliability checks leave gaps in task coverage and validation.

Evaluation Scope and Model Coverage

  • MMGR evaluates zero-shot models without task-specific training, so its scores may underestimate performance after task adaptation.Some recent systems may also be absent, and video-to-video editing is outside the evaluated text-to-image and text-to-video scope.Source: Limitations · p. 10

Deterministic Evaluation Limitations

  • Physical Commonsense and several embodied-navigation tasks rely only on VLM evaluation, whose spatial-reasoning and physical-understanding weaknesses affect those scores.Maze and Sudoku use exact pixel-based and OCR-based checks.Source: Limitations · p. 10

Benchmark Design Constraints

  • Human-designed difficulty tiers may not match model-specific difficulty, while Sudoku and ARC floor effects limit discrimination among weaker video models.The benchmark also excludes continuous control, long-horizon planning, and open-ended creative generation; standardized prompts favor consistency over per-model optimization.Source: Limitations · p. 10

Generalization and Ecological Validity

  • MMGR’s structured tasks and clear ground-truth answers may underrepresent ambiguous, partially observable, subjective, or open-ended real-world settings.Controlled tasks may omit distribution-matched scenarios on which video models perform well.Source: Limitations · p. 10

Reliability Scope

  • Reliability checks use sampled manual annotations rather than exhaustive labels, and raw accuracy can be inflated by class imbalance.Because annotating every output is infeasible, assessment relies on automatic metrics, especially for VLM-only navigation and physical-commonsense tasks.Source: Limitations · p. 10

Ethical Considerations

MMGR is presented as a diagnostic benchmark for reasoning in generative models, with human annotation and controlled maze tasks supporting its evaluation. The accompanying procedures emphasize inference-time assessment, verifiable criteria, and frame-level inspection of outputs and failures.

Ethical Considerations

  • MMGR is a diagnostic framework for surfacing current systems’ limitations and failure modes, not a new modeling or deployment technique.The authors connect auditable outcomes to applications such as decision-making, education, and scientific reasoning, while noting possible misuse and over-reliance risks.Source: Ethical Considerations · pp. 10–11

A Computational Experiments; A.1 Model Size and Budget

B Human Subjects Including Annotators

  • Three graduate-level annotators independently scored balanced subsets, with each output judged by at least two people and disagreements resolved through discussion.They were paid $25 per hour and participated with informed consent.Source: B Human Subjects Including Annotators · pp. 18–19

B.1 Instructions Given to Participants

  • Annotators applied binary labels to applicable rubric criteria and inspected videos frame by frame, judging literal task constraints rather than visual quality.Overall success required every applicable criterion to hold.Source: §B.1 · p. 19

C.1 Task Description; C.2 Hard-Level Control

  • The 2D Maze task probes spatial topology, logical planning from start to goal, and sequential temporal execution, while evaluation cases systematically vary maze structure and difficulty through generator and start–goal-placement parameters.It uses DFS and Wilson’s Algorithm and four start–goal placement schemes: corner-to-corner, corner-to-random, random-to-corner, and random-to-random.Source: §C.1 · p. 19, §C.2 · p. 19

C.3 Evaluation and Metrics

  • Maze evaluation combines checks for layout changes, wall crossings, exploration, and target achievement; overall success requires an unchanged maze, no wall crossing, and reaching the target.The evaluator receives the generated output, a ground-truth solution image, and a structured prompt.Source: §C.3 · pp. 20–21

C.4 Case Study; C.5 Evaluation Results

  • Maze case studies contrast static-image path artifacts with videos that can preserve valid paths but may also alter walls or violate constraints between frames.The reported analysis distinguishes Veo-3’s result-mimicking paths from Sora-2’s visible backtracking with incoherent maze structures; human review finds transient violations that automated VLM metrics can miss.Source: §C.4 · p. 21, §C.5 · p. 21

C.5.1 VLM-Based Evaluation

This section evaluates maze and Sudoku generation using VLM- and pixel-based criteria, examining goal-reaching, maze structure, physical constraints, puzzle completion, and solution validity. Its evaluations and case studies assess whether image and video generators sustain constraint-aware reasoning, including the finding that stronger Sudoku completion accuracy does not necessarily yield a valid solution.

C.5.1 VLM-Based Evaluation

  • Veo-3 leads video generators at 38.72%–51.50% overall, while Sora-2 scores 2.50%–10.00% and Wan-2.2 scores 0.83%–5.00%.Veo-3’s near-zero Action Reflection (0.00%–2.50%) accompanies high Target Achievement (50.78%–61.50%) but Cross Wall rates of 15.62%–25.64%; Sora-2 explores more (40.00%–77.50% Action Reflection) but changes mazes (45.00%–67.50%) and rarely reaches targets (10.00%–25.00%), while Wan-2.2 has 79.17%–90.83% Cross Wall rates.Source: §C.5.1 · pp. 21–23
  • GPT-image-1.5 has the best image-model overall scores on DFS mazes (3.33%–13.33%), while Qwen-image scores 20.00% on Wilson’s Easy mazes.GPT-4o-image reaches targets in 72.50%–90.00% of cases but scores near zero overall after changing the maze structure in 60.00%–95.00% of cases; across modalities, high target achievement alone does not ensure success, and image models degrade more sharply with difficulty than video models.Source: §C.5.1 · p. 23

C.5.2 Limitations and Insights from VLM-Based Evaluation

  • AutoEval, instantiated as Gemini-2.5-Pro, can miss transient Cross Wall and Maze Changed violations on Hard tasks, causing systematic performance overestimation.Visualized plans can simplify evaluation into a spatial path check when grounded, but Veo-3’s hallucinated blue trajectories obscure its actual reasoning and further confuse the evaluator.Source: §C.5.2 · p. 23

C.5.3 Pixel-based Evaluation

  • Pixel-based evaluation uses classical computer vision to compare generated mazes with ground-truth solutions, enabling fine-grained, reproducible, cost-free assessment without VLMs or LLMs.For videos, it reconstructs the agent’s traversal path and checks layout across frames, wall crossings, path matching, target achievement, and excessive exploration; the gated overall score is 1 only when all required criteria pass.Source: §C.5.3 · pp. 23–24
  • Pixel-based evaluation is much stricter: the best overall score is 30.00% for Nano-banana Pro on DFS Medium, and Veo-3 is the only video model with nonzero scores, limited to Easy mazes.Veo-3 scores 10.50% on DFS Easy and 11.50% on Wilson’s Easy, but its Medium and Hard performance collapses as Cross Wall rates rise to 96–99% and Maze Changed reaches 100%.Source: §C.5.3 · pp. 24–25

D Sudoku

  • Sudoku tests whether a model can satisfy row, column, and subgrid constraints through logical inference to produce a valid complete grid.Source: D Sudoku · p. 26

D.1 Hard-Level Control

  • The Sudoku set contains 300 puzzles, balanced across Easy, Medium, and Hard levels and 4×4 and 9×9 grids, with difficulty controlled by grid size and clue count.Each difficulty level has 100 puzzles: 50 of each grid size; Hard puzzles use fewer clues while retaining a unique solution.Source: §D.1 · p. 26

D.2 Evaluation and Metrics

  • Gemini-2.5-Pro evaluates generated Sudoku images or videos against the puzzle and ground-truth solution, with overall success requiring unchanged clues, zero constraint violations, and complete accuracy.The evaluation also records clue changes, constraint violations, completion accuracy, and—for video only—whether actions show explicit reflection.Source: §D.2 · pp. 27–28

D.3 Case Study

  • Nano-banana can solve an easy 4×4 Sudoku while preserving clues and valid steps, but its failures include missing or repeated digits, constraint violations, and clue changes.The case studies conclude that both Nano-banana and Veo-3 show surface-level competence without maintaining a robust constraint-aware trajectory throughout the solution.Source: §D.3 · pp. 28–29
  • Veo-3 usually starts Sudoku in the top-left box and makes self-reflective edits that can either resolve conflicts or overwrite correct digits and invalidate the solution.Its early frames may preserve clues, but later frames often modify clues or oscillate between partial solutions, indicating temporal drift and weak persistent constraint representation.Source: §D.3 · p. 29

D.4.1 VLM-Based Evaluation

  • 22.67% on Easy, 10.67% on Medium, and 7.33% on Hard are GPT-image-1.5’s overall success rates on 4×4 Sudoku, while Veo-3 leads video models at 1.20% on Easy.Sora-2 and Wan-2.2 score 0% overall on 4×4 puzzles.Source: §D.4.1 · p. 29

D.4 Evaluation Results

  • 97% completion accuracy, 1.33% constraint violations, and 35% clue changes leave Nano-banana Pro with 0% overall success on Easy 4×4 Sudoku.Across models, strict success requires avoiding both clue changes and constraint violations; video models’ sequential generation compounds errors despite high action reflection.Source: §D.4.1 · pp. 29–30, §D.4 · p. 29

D.4.1 VLM-Based Evaluation

  • 4.26% on Easy and 7.14% on Hard are Sora-2’s reported 9×9 overall scores, but 66%–100% clue changes and 92%–100% constraint violations make this apparent edge a VLM evaluation inconsistency, not genuine superiority.Image models score 0% overall, despite Qwen-image’s 72%–74% completion accuracy and 5%–10% constraint violations.Source: §D.4.1 · pp. 29–30

D.4.2 OCR-Based Evaluation

This section applies deterministic OCR-based evaluation to Sudoku and describes ARC task construction, classification, scoring, and results. It reports universal 9×9 Sudoku failure and an image–video performance gap on ARC, alongside video context drift.

D.4.2 OCR-Based Evaluation

  • OCR reconstructs Sudoku grids and checks clues, constraints, and completion; video evaluation also inspects frames for gradual solving behavior.The analysis identifies clue modification and constraint violations as recurring failures, with performance collapsing as grids scale from 4×4 to 9×9.Source: §D.4.2 · pp. 30–32

D.4.2 OCR-Based Evaluation; E.4 Evaluation Results

E Abstraction and Reasoning Corpus (ARC)

E.1 Hard-Level Control

  • The curated ARC suite contains 456 tasks: 381 from ARC v1 and 75 newer, structurally complex tasks from ARC v2.Source: §E.1 · p. 33

E.1.1 Two-Level Classification System

  • ARC cases are classified by whether input and output shapes match, then assigned Easy, Medium, or Hard levels using five grid-level features.Source: §E.1.1 · p. 33

E.2 Evaluation and Metrics

  • Gemini-2.5 Pro evaluates generated outputs; exact ground-truth matching is the primary metric, with pattern recognition, grid integrity, and color accuracy as diagnostics.Source: §E.2 · p. 34

E.3 Case Study

  • Video models altered the static ARC demonstrations across frames, including color shifts, pattern deformation, and erased grid contents.Source: §E.3 · p. 35

E.4 Evaluation Results

  • Video-model context drift corrupts the task examples, while the strongest video model, Sora-2, scores 11.67% on ARC against Nano-banana Pro’s 28.07%.Source: §E.4 · p. 35

E.4.1 VLM-Based Evaluation

The ARC and visual-math analyses show that generated outputs can preserve some visible structure or reach plausible answers without robust task correctness or valid intermediate reasoning.

E.4.1 VLM-Based Evaluation

  • On ARC, Nano-banana Pro leads at 28.07%, versus GPT-image-1.5 at 12.72% and Sora-2 at 11.67%, while its v1/v2 scores remain stable at 30.54% and 30.36%.Sora-2 falls from 20.18% to 1.33% between versions, while Nano-banana Pro remains stable.Source: §E.4.1 · pp. 35, 37

E.4.2 Key Findings and Implications

  • ARC video generations can corrupt fixed demonstrations across frames, while strong pattern and grid scores may still fail to yield color accuracy and overall correctness.Color Accuracy is only 20–25% of Grid Integrity across models; the cascade suggests precise color execution is a bottleneck.Source: §E.4.1 · p. 37, §E.4.2 · p. 38

F Math

  • Visual Math tests visual mathematical understanding and deduction, including temporal reasoning through videos that animate steps from premises to solution.Source: F Math · p. 38

F.1 Hard-Level Control

  • The evaluation spans GSM8K, MATH, AIME 2024, AIME 2025, and Omni-MATH, with Omni-MATH also stratified by five difficulty levels and eight categories.Source: §F.1 · p. 38

F.2 Evaluation and Metrics

  • The metrics separate final correctness, valid intermediate reasoning, and video-only self-correction; complete Overall Score requires both final and intermediate correctness.Source: §F.2 · p. 39

F.3 Evaluation Results

  • Across final math scores, Gemini-3-Pro leads at 74.14%, followed by Nano-banana Pro at 72.69%, while Wan-2.2 scores 0.00%.Veo-3 and Sora-2 are near 10%.Source: §F.3.1 · p. 39

F.3.1 Overall Performance Patterns

  • On GSM8K, Veo-3 reaches 74.00% Outcome Success but only 12.00% Process Success, exposing a gap between correct answers and valid intermediate reasoning.Nano-banana Pro achieves 97.83% on both metrics.Source: §F.3.1 · pp. 39–40
  • Video models’ Action Reflection scores remain at 0–16%, and their performance varies by difficulty, with Sora-2 stronger on easier tasks and Wan-2.2 on AIME 24.Sora-2 reaches 30.00% on GSM8K; Wan-2.2 reaches 15.56% on AIME 24.Source: §F.3.1 · p. 40

F.3.2 Dataset Difficulty Progression

The section evaluates model performance across mathematical reasoning and embodied-navigation benchmarks, covering variation by difficulty and domain alongside navigation task settings, verification methods, and final-score summaries.

F.3.2 Dataset Difficulty Progression

  • Nano-banana Pro scores 97.8% on GSM8K and 91.84% on MATH500, while Veo-3’s GSM8K outcome success (74.00%) exceeds its process success (12.00%) and it reaches 18.00% on MATH500.Sora-2 records 30.00% Overall on GSM8K and remains stable at 31.91% on MATH500, while GPT-4o-image declines from 75.65% to 27.34%.Source: §F.3.2 · pp. 40–41
  • Nano-banana Pro reaches 66.67% on AIME25 and 63.06% on Omni-MATH; Veo-3 leads video models on Omni-MATH at 3.89%, over Sora-2 (0.62%) and Wan-2.2 (0.41%).Wan-2.2 reaches 15.56% on AIME24 despite failing on easier tasks.Source: §F.3.2 · p. 41

F.3.3 Omni-MATH Deep Dive

  • Omni-MATH difficulty scaling is non-monotonic: Veo-3 scores 6.06% on T0, 1.47% on T3, and 5.00% on hardest T4, while Nano-banana Pro rises from 26.67% on T1 to 82.76% on T4.The paper suggests canonical high-difficulty templates may be easier to reproduce than ambiguous intermediate problems; Wan-2.2 also performs better on the hardest benchmarks than intermediate ones.Source: §F.3.3 · p. 41, §F.3.4 · p. 42
  • Video models score 8.33% in Geometry but 0.00% Overall in Calculus, Discrete Math, Applied Math, and Precalculus, while Nano-banana Pro reaches 53.33% in Calculus and 68.75% in Applied Math.Veo-3’s Precalculus Outcome Success is 21.74% despite 0.00% Overall, indicating answer-process dissociation.Source: §F.3.3 · p. 41

F.3.4 Key Findings and Implications

  • Video reasoning incurs a temporal penalty, with hard-benchmark gaps often exceeding 4–6×, and Veo-3’s 74.00% GSM8K outcome success contrasts with 12% process success.Case studies attribute the disconnect to temporal drift, including invalid answer transitions in 62% of cases, harmful self-corrections, and center-biased attention that misses peripheral constraints.Source: §F.3.4 · p. 42, §F.3.5 · p. 42

G.1 Task Definition

  • Embodied Navigation tests spatial understanding, grounding, temporal reasoning, physical commonsense, and multi-step planning across panoramic last-mile, top-down, 3D real-world, and SLAG settings.The settings respectively probe short-range wide-field navigation, global path planning, multi-room and multi-level layouts, and joint localization with scene generation.Source: G Embodied Navigation · p. 43, §G.1 · p. 43

G.2 Hard-Level Control

  • The navigation dataset controls difficulty along environmental complexity, view fidelity, trajectory distance, and destination specification, with task-specific measures designed to preserve cross-task comparability.Complexity varies scene structure; fidelity is rated 3–5; long trajectories include a significant turn; destinations may be visually marked or described with disambiguating landmarks and floor identifiers.Source: §G.2 · p. 43, §G.2.1 · p. 44, §G.2.2 · p. 44

G.3.4 Holistic Performance

  • Navigation evaluation combines geometric task-completeness, physical-understanding, and instruction-following metrics, while Overall success requires every fine-grained metric to pass simultaneously.The binary criteria cover reaching or approaching the destination, physically plausible and continuous motion, and preservation of the destination and static scene.Source: §G.3 · p. 45, §G.3.1 · p. 45, §G.3.2 · p. 46

G.3.5 Geo-Align

  • Geo-Align reconstructs generated trajectories with VGGT and aligns them to Habitat shortest paths using cubemap conversion, scale normalization, and XZ-plane heading rotation.It supports physical-faithfulness verification for panoramic last-mile navigation.Source: §G.3.5 · p. 47
  • Geo-Align assesses task success with SR and SPL (Anderson et al., 2018; Zhang et al., 2024) and path fidelity with nDTW and sDTW (Ilharco et al., 2019).SPL accounts for route efficiency, while nDTW and sDTW measure trajectory similarity to the intended path.Source: §G.3.5 · p. 47
  • It also measures trajectory drift with ATE and RPE, compares predicted and ground-truth path lengths, and evaluates terminal distance and heading errors.These checks expose looping or premature stopping as well as endpoint imprecision.Source: §G.3.5 · p. 47

G.3.6 View-Syn

  • For static-view tasks, View-Syn compares generated sequences with fixed-camera ground truth using FVD for video or final-frame FID for images, normalized so higher scores indicate better alignment.The score is a high-level proxy for temporal coherence and semantic goal verification when explicit pose extraction is unavailable.Source: §G.3.6 · pp. 47–48
  • View-Syn’s five metrics cover video distribution alignment with FVD, image distribution and quality with FID, and structural, perceptual, and reconstruction quality with SSIM, LPIPS, and PSNR.LPIPS uses deep features to approximate human perceptual judgments.Source: §G.3.6 · p. 48

G.4 Overall Evaluation Results

  • Table 26 reports one final score per embodied-navigation task across the four embodied rows in Table 4; later detailed tables are diagnostic breakdowns for available model subsets.The passage provides no score values or model comparisons.Source: §G.4 · p. 48

G.4.1 Comparative Analysis of Generators

Nano-banana Pro leads all four embodied navigation tasks, while the broader findings show that video models’ spatial perception does not reliably translate into goal-directed navigation.

G.4.1 Comparative Analysis of Generators

  • Nano-banana Pro leads all four tasks: Last-Mile Navigation (75.83%), Top-down View Navigation (33.05%), 3D Real-World Navigation (85.00%), and SLAG (37.29%).Veo-3 is competitive on Last-Mile Navigation but falls on Top-down View and SLAG, with video models struggling to preserve goal state, geometry, and cross-view alignment.Source: §G.4.1 · p. 48

G.5 Key Observations

  • Video models show strong egocentric spatial perception and latent cross-view alignment, yet free-form long-horizon navigation remains unreliable; structured SLAG conditioning elicits alignment without ensuring navigation.Their profile includes semantic grounding but poor rule-following, goal-directed motion, and robustness to environmental complexity.Source: §G.5 · p. 49

H.1 Task Definition

  • Last-Mile Navigation isolates short-horizon movement toward a visible goal from one panoramic RGB observation, combining geometric localization with semantic target recognition.The target may be marked or specified by object class.Source: §H.1 · p. 50

H.2 Evaluation and Metrics

  • Binary pass/fail metrics assess plausible motion, structural consistency, and destination semantics using automatic VLM or human judgments on rollout traces and generated frames.Source: §H.2 · p. 50

H.2.1 Task Completeness Metrics (Geometry Only)

  • Success Score 3D checks whether the final position reaches the target, while Oracle Success Score 3D credits reaching it at any trajectory timestep.Both scores isolate geometric accuracy from visual fidelity and physical consistency.Source: §H.2.1 · pp. 50–51

H.2.2 Gate Metrics and Holistic Performance

  • Gate Metrics combine geometric completion with semantic and physical validity; Overall Success requires all seven binary checks to pass.Original-destination scoring blocks fabricated targets or warped layouts from counting as success.Source: §H.2.2 · p. 51

H.3 Evaluation Results

  • In the diagnostic subset, Veo-3 and Nano-banana function while Sora-2 and GPT-4o fail; Veo-3’s auto-metric success is 73.33% versus 25.00% under human review.Auto-evaluators track camera motion but fail to penalize physical hallucinations and logic violations.Source: §H.3 · p. 51

H.3.1 VLM-Based Evaluation

These navigation evaluations show that apparent path success can diverge from end-to-end validity: automatic scoring can exceed human judgment, and text-described destinations are harder than visual markers. Task-specific gates and case studies distinguish reaching a goal from preserving the scene, agent identity, and physical constraints.

H.3.1 VLM-Based Evaluation

  • In the selected panoramic subset, Veo-3 and Nano-banana navigate successfully, while Sora-2 and GPT-4o-image score 0.00% overall.Nano-banana is more resilient on long routes, complex floors, and text-described destinations; its success rises with visual fidelity, unlike Veo-3's plateau.Source: §H.3.1 · pp. 51–52

H.4 Case Study: Qualitative Failure Modes

  • Figure 21 contrasts Veo-3's duplicate-agent failure, Sora-2's physical and geometric violations, and Wan-2.2's unstable environment despite smooth motion.Veo-3 loses agent identity; Sora-2 remaps the target and clips through railings; Wan-2.2 erases a door and hallucinates pillars, preventing successful navigation.Source: §H.4 · pp. 52, 54, Figure 21 · p. 53

I.1 Task Definition

  • Top-down navigation tests trajectory planning through partially occluded layouts and grounding goals described at varying levels of abstraction.The task combines spatial planning with semantic understanding in complex, human-centric environments.Source: §I.1 · p. 54

I.2 Evaluation and Metrics

  • Top-down evaluation retains the shared physical-understanding and instruction-following checks, then forms gated holistic scores from task-specific completeness metrics.The shared checks include object semantics, agent consistency, spatial alignment, destination integrity, and scene consistency.Source: §I.2 · p. 54

I.2.1 Task Completeness Metrics (Geometry Only)

  • The 2D Success Score requires the agent to stop inside the goal, whereas Oracle Success credits encountering the goal during the rollout.These geometry-only measures assess destination accuracy rather than visual fidelity or physical consistency.Source: §I.2.1 · p. 54

I.2.2 Gate Metrics and Holistic Performance

  • Overall Success requires all seven binary checks; destination and physical-validity gates reject altered layouts, object clipping, teleportation, and unnatural drift.The original-destination gate combines goal arrival, destination integrity, and scene consistency; Physical Validity combines object semantics, agent consistency, and spatial alignment.Source: §I.2.2 · p. 55

I.3 Evaluation Results

  • Veo-3's automatic success is 37.14% versus 10.34% by human judgment; text-described goals also produce a >3× performance drop.The text-description drop is relative to visual-marker instructions.Source: §I.3 · p. 55

I.3.1 VLM-Based Evaluation

The 3D navigation results expose a gap between raw goal arrival and holistic validity, while task-specific criteria check geometry, scene integrity, and physical behavior. In the top-down evaluation, results also vary with instruction form and environmental complexity, and qualitative cases show distinct control and scene failures.

I.3.1 VLM-Based Evaluation

  • On floor01, Veo-3 reaches 65.71% Success Score (2D), versus 41.67% for Nano-banana and 22.81% for Sora-2.Veo-3's Spatial Alignment Score is 91.43%; the authors attribute its trajectory-coherence advantage to temporal attention. As environments become more complex, its Overall Success drops from 37.14% to 13.89%, while higher fidelity improves it from 20.83% at quality03 to 29.17% at quality04.Source: §I.3.1 · pp. 55–56
  • For text-described targets, Veo-3 reaches 11.43% success and Nano-banana 8.33%; Nano-banana's Object Semantic Score is 88.89% in simple environments.The results contrast Nano-banana's static recognition strength with Veo-3's stronger trajectory dynamics.Source: §I.3.1 · pp. 55–56

I.4 Case Study: Qualitative Failure Modes

  • Figure 23's cases distinguish Veo-3's post-arrival drift, Sora-2's scene replacement and obstacle clipping, and Wan-2.2's table-as-agent confusion and shifted target.Veo-3 enters the goal but continues beyond it; Sora-2 loses scene integrity, while Wan-2.2 preserves some layout but misidentifies the controllable agent.Source: §I.4 · pp. 56, 58, Figure 23 · p. 57

J.1 Task Definition

  • 3D real-world navigation tests egocentric interpretation of scanned environments, valid action trajectories, and goal identification from abstract or textual descriptions.The task probes 3D spatial reasoning and visual-semantic grounding.Source: §J.1 · p. 58

J.2 Evaluation and Metrics

  • The 3D task uses binary VLM- or human-labeled rollout and frame checks, retaining the shared physical-understanding and instruction-following criteria.The checks cover object semantics, agent consistency, spatial alignment, destination integrity, and scene consistency.Source: §J.2 · p. 58

J.2.1 Task Completeness Metrics (Geometry Only)

  • The 3D Success Score requires stopping inside the destination volume, whereas Oracle Success credits reaching its vicinity at any point.Both are geometric measures; visual fidelity and physical plausibility are enforced by gate metrics.Source: §J.2.1 · p. 58

J.2.2 Gate Metrics and Holistic Performance

  • 3D gates require destination and scene integrity alongside physical validity, blocking shortcuts through altered layouts or implausible motion; Overall Success conjuncts all seven checks.The gates address failure modes such as teleporting between floors, cutting through occluded rooms, and hallucinating nearer goals.Source: §J.2.2 · p. 59

J.3 Evaluation Results

  • Video models reach raw success rates of 85–89% in simple navigation but score 0–3% on holistic metrics, while Nano-banana maintains scene consistency above 95% and physical validity.The reported video-model failures involve scene hallucination and physics violations.Source: §J.3 · p. 59

J.3.1 VLM-Based Evaluation

This section evaluates generated outputs across embodied navigation and physical-reasoning scenarios, including spatial and cross-view reasoning, linguistic grounding, synchronized representations, SLAG’s joint 3D navigation and 2D map generation, physical concepts, and sports. It also describes physics-aware VLM evaluation of generated videos using binary criteria across simple, complex, and chain-reaction interactions.

J.3.1 VLM-Based Evaluation

  • In the diagnostic navigation subset, Nano-banana dominates video models, sustaining 75–87.5% Overall Success with 86–100% Scene Consistency and 94–100% Physics Validness.Its success rises from 75% on floor01 to 83.33% on floor02plus, while Veo-3 falls to 0% in complex multi-floor environments.Source: §J.3.1 · p. 59
  • Sora-2 scores better on spatial alignment and scene consistency but nearly always misses the original destination; Veo-3 reaches it more often while violating scene and physical consistency.GPT-4o-image recognizes objects well (≈80–90%) but has low navigation success (≈13–17%).Source: §J.3.1 · pp. 59–60

J.4 Case Study: Qualitative Failure Modes

  • In the three-model humanoid-robot navigation case, Veo-3 alone reached the destination (Oracle Success Score = 1), despite replacing the curved stairs and jumping from the second floor to the ground, clipping into the floor.The task required a humanoid robot to reach a specific red target zone while following physical constraints, step-by-step stair climbing, and a consistent dollhouse isometric view.Source: §J.4 · p. 60

K.1 Task Definition

  • SLAG couples 3D navigation with a generated 2D trajectory, evaluating task success in both views, trajectory alignment, and physical and scene consistency through joint gates.The 2D map is supplied without a fixed destination, so trajectory alignment and a 3D destination gate help expose cross-view mismatches such as hallucinated corridors or 3D teleportation.Source: §K.1 · p. 62, §K.2 · p. 62, §K.2.1 · p. 62

K.3.1 VLM-Based Evaluation

  • Nano-banana leads SLAG on floor01, with a 27.78% Holistic Metric and 88.89% Trajectory Alignment, while video baselines score 11.86%/10.34% holistic and 44.07%/55.17% alignment.Despite respectable Physics Validness, video models’ Success (3D) Original Destination is 18.64% for Veo-3 and 22.41% for Sora-2, below Nano-banana’s 38.89%.Source: §K.3.1 · p. 64
  • Textual destination descriptions sharply reduce success across model families, while increasing environmental complexity produces divergent holistic outcomes for image and video baselines.Nano-banana’s 3D Success falls from 81.25% to 17.65% and Holistic from 50.00% to 8.82% on text descriptions; GPT-4o-image, Veo-3, and Sora-2 show similar drops. Across floors, Nano-banana’s Holistic rises from 27.78% to 30.00%, Sora-2’s from 10.34% to 15.52%, while GPT-4o-image falls from 25.00% to 6.90%.Source: §K.3.1 · p. 64

K.4 Case Study: Qualitative Failure Modes

  • In SLAG case studies, Veo-3 reaches the 3D goal but its map trace drifts, Sora-2 hallucinates the scene and clips through objects, and Wan-2.2 teleports and fails in both views.The qualitative examples show that visual fluency does not ensure semantic linkage between split-screen representations over time.Source: §K.4 · pp. 64, 66

L.1 Data Sources and Task Structure

  • Physical Commonsense tests physical principles and causal dynamics in video through two complementary axes: VideoPhy-based interactions such as collisions and fluid behavior, and sports prompts requiring compositional reasoning.The task covers gravity, momentum, collisions, and material properties, and examines 3D spatial interactions and temporal cause-and-effect (Yi et al., 2020).Source: L Physical Commonsense · p. 66, §L.1 · p. 66

L.3 Evaluation and Metrics

  • Gemini-2.5-Pro scores videos 0/1 on physics accuracy, motion quality, visual realism, and prompt adherence, with Overall Success requiring compliance on all four dimensions.The criteria respectively assess physical laws and plausible interactions, temporal coherence, visual fidelity, and semantic match to the prompt.Source: §L.3 · p. 67

L.4 Case Study

  • Veo-3’s parachute-inflation example scores fully across all four dimensions, while coffee grinding and ballet-turn examples expose failures in causal material change and angular-momentum dynamics.Coffee beans change abruptly into uniform grounds without mechanical cause, while the dancer rotates without the whipping motion needed to generate momentum; both failures violate physical expectations despite plausible-looking frames.Source: §L.4 · pp. 67–69

L.5 Evaluation Results

  • Sora-2 leads the physical subsets with 70.00% average success, ahead of Veo-3 (51.02%) and Wan-2.2 (24.00%), while Wan-2.2 reaches 84–96% Visual Realism despite 24% aggregate Overall.Sora-2 scores 76.00% on Physical Concepts and 64.00% on Sports; Sports scenarios are generally easier than Physical Concepts, and Veo-3 achieves 0% success on rigid-body collisions versus 75% on Solid-Fluid interactions.Source: §L.5 · p. 70

L.5.1 VLM-Based Evaluation

On Physical Commonsense, Sora-2 leads the evaluated video generators, but strong visual realism does not guarantee physical correctness. Models also show distinct difficulty patterns across sports, physical interactions, and prompt difficulty.

Evaluation setup

  • Table 36 evaluates Veo-3, Sora-2, and Wan-2.2 on Physical Concepts and Sports Scenarios using Gemini-2.5-Pro; the sample set balances fundamental understanding and compositional applications.The Average row is a diagnostic aggregate; Table 4 reports the two task-level rows separately.Source: §L.5.1 · p. 70, Table 35 · p. 92, Table 36 · p. 92

Overall Performance Trends

  • 70.00% aggregate success rate puts Sora-2 ahead of Veo-3 (51.02%) and Wan-2.2 (24.00%), despite all models’ high visual-realism scores.Visual Realism scores range from 84–92%, while Physics Accuracy and Prompt Adherence vary more widely.Source: §L.5.1 · p. 70, Table 36 · p. 92

Scenario-Specific Difficulty

  • Sports Scenarios are easier than Physical Concepts across models: Veo-3 rises from 41.67% to 60.00%, while Wan-2.2 reaches only 21.33% on Sports.The authors suggest models benefit from learned biomechanical patterns in human-centric activities but struggle with fine-grained material interactions.Source: §L.5.1 · p. 70

Sport-Specific Patterns

  • Ballet is hardest across models (33–44% success), while Swimming reaches 83% for Veo-3 and Sora-2; Sora-2 is more consistent across difficulty levels.Wan-2.2 scores 0% on Diving but 44.4% on Ballet, and falls from 14–33% across difficulty levels.Source: §L.5.1 · p. 70, Table 37 · p. 92

States-of-Matter Analysis

  • Rigid-body collisions expose divergent performance: Veo-3 scores 0% on Solid-Solid interactions, while Sora-2 reaches 100%.Wan-2.2 scores 83.3% on Fluid-Fluid interactions, but falls from 35.4% on Easy to 11.1% on Hard cases.Source: §L.5.1 · p. 70, Table 38 · p. 92

The Visual–Physical Disconnect

  • On Sports Scenarios, Wan-2.2 pairs the highest Visual Realism (96.00%) with the lowest Physics Accuracy (42.67%) and Prompt Adherence (21.33%).The authors interpret this as visual plausibility without corresponding physical consistency.Source: §L.5.1 · p. 70

Related papers

Loading 2512.14691v3…