PaperlayerView exact revision on arXiv

Source-linked AI summary

ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation

Yongyuan Liang, Wei Chow, Feng Li, Ziqiao Ma, Xiyao Wang, Jiageng Mao, Jiuhai Chen, Jiatao Gu, Yue Wang, Furong Huang

arXiv:2511.01163v1cs.CV

TL;DR

Existing evaluations largely assess language and visual abilities separately, leaving reciprocal cross-modal reasoning under-tested. ROVER is a human-annotated benchmark of 1,312 tasks grounded in 1,876 images, evaluated alongside 17 unified models. Interleaved models outperform non-interleaved counterparts in reasoning-dependent visual generation, while models show a gap between physical and symbolic visual reasoning.

  • Problem

    Prevailing evaluations treat multimodal understanding and generation in isolation, emphasizing language reasoning in textual benchmarks and pixel outcomes in visual benchmarks.

    Source: §1 · p. 1

  • Method

    ROVER is a human-annotated benchmark of 1,312 tasks grounded in 1,876 images, testing verbal reasoning for image generation and visual intermediates for text generation.

    Source: §1 · p. 2

  • Results

    Interleaved models significantly outperform non-interleaved counterparts in reasoning-dependent visual generation, while strong unimodal models cannot replicate this cross-modal reasoning when combined.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results identify reciprocal cross-modal reasoning as a critical role in holistic omnimodal generation that independent optimization of constituent modalities does not fulfill.

    Source: §1 · p. 3

    Current unified models remain constrained by difficulty representing abstract concepts visually, limiting cross-modal reasoning on logic-intensive problems.

    Source: §4.4 · p. 12

Abstract

from arXiv · show

Unified multimodal models (UMMs) have emerged as a powerful paradigm for seamlessly unifying text and image understanding and generation. However, prevailing evaluations treat these abilities in isolation, such that tasks with multimodal inputs and outputs are scored primarily through unimodal reasoning, i.e., textual benchmarks emphasize language-based reasoning, while visual benchmarks emphasize reasoning outcomes manifested in the pixels. We introduce ROVER to address this pressing need to test reciprocal cross-modal reasoning, the use of one modality to guide, verify, or refine outputs in the other, an ability central to the vision of unified multimodal intelligence. ROVER is a human-annotated benchmark that explicitly targets reciprocal cross-modal reasoning, which contains 1312 tasks grounded in 1876 images, spanning two complementary settings. Verbally-augmented reasoning for visual generation evaluates whether models can use verbal prompts and reasoning chains to guide faithful image synthesis. Visually-augmented reasoning for verbal generation evaluates whether models can generate intermediate visualizations that strengthen their own reasoning processes for question answering. Experiments on 17 unified models reveal two key findings: (i) Cross-modal reasoning determines visual generation quality, with interleaved models significantly outperforming non-interleaved ones; notably, combining strong unimodal models fails to achieve comparable reasoning. (ii) Models show dissociation between physical and symbolic reasoning: they succeed at interpreting perceptual concepts literally but fail to construct visual abstractions for symbolic tasks, where faulty reasoning harms performance. These results highlight reciprocal cross-modal reasoning as a critical frontier for enabling true omnimodal generation.

1. Introduction

Prevailing evaluations largely assess language and visual abilities separately, leaving reciprocal cross-modal reasoning insufficiently tested. ROVER benchmarks that capability in image and text generation, and evaluation of 17 unified models reveals substantial reasoning gaps.

1. Introduction

  • Existing evaluations emphasize language reasoning or visual outcomes, rather than whether one modality guides, verifies, or refines outputs in the other.Text-only metrics can miss visual fidelity, while image-only metrics cannot verify valid reasoning.Source: §1 · pp. 1–2
  • ROVER is a human-annotated benchmark of 1,312 tasks grounded in 1,876 images, covering verbal reasoning for image generation and visual reasoning for text generation.The two settings include multiple reasoning domains and subtasks, with interleaved visual intermediates in the text-generation setting.Source: §1 · p. 2, Figure 1 · p. 2
  • Its evaluation combines an automated VLM judge with expert validation, scoring reasoning coherence, output alignment, and consistency between intermediate steps and final outputs.Visual-generation tasks also use established image consistency and quality metrics.Source: §1 · p. 2
  • Interleaved models significantly outperform non-interleaved counterparts in reasoning-dependent visual generation, while combining strong unimodal models does not replicate their cross-modal reasoning.Models also interpret perceptual and physical concepts more successfully than symbolic visual abstractions, where faulty reasoning can degrade downstream performance.Source: §1 · p. 3
  • The paper contributes a task taxonomy, verification-ready instances with process targets and visual artifacts, and a multidimensional scoring protocol.It evaluates 17 unified models to identify limitations in cross-modal reasoning.Source: §1 · p. 3

2. Related Work

Unified multimodal models use varied architectures to combine comprehension and generation, but benchmarks have not comprehensively assessed how those capabilities coordinate. ROVER focuses on the interplay between reasoning and generation and on intermediate reasoning processes.

2. Related Work

  • Unified multimodal architectures integrate comprehension and generation through visual-token prediction, diffusion or flow-matching heads, or routing features to external image generators.Some approaches keep a pretrained reasoning backbone frozen while routing its intermediate features through learnable queries.Source: §2 · pp. 3–4
  • Existing benchmarks inadequately assess whether multimodal understanding, reasoning, and generation mutually enhance one another or suffer coordination deficiencies.The comprehensive evaluation of these relationships remains largely unexplored.Source: §2 · p. 4
  • Prior reasoning-guided image benchmarks often overlook intermediate reasoning quality and its alignment with generated images, while some focus on similarity to human-provided ground truth.ROVER investigates the interplay between reasoning and generation.Source: §2 · p. 4

3. ROVER Benchmark

ROVER comprises two complementary benchmarks: verbal reasoning that guides image generation and visual intermediates that support verbal reasoning. Both use curated tasks and evaluation protocols designed to assess reasoning as well as outputs.

3.1. Verbally-Augmented Reasoning for Visual Generation

  • ROVER-IG spans natural science, culture and art, common sense, and logic and math, with subtasks for temporal, spatial, causal, imaginative, quantitative, puzzle, and geometry reasoning.Tasks use complex text-driven reasoning chains to guide image synthesis.Source: §3.1 · p. 5
  • ROVER-IG contains 908 human-verified visual-generation tasks across four domains and seven reasoning subtasks, including single- and multi-image scenarios.Experts and language models collaboratively generate reasoning tasks from selected images; each task includes a prompt, target descriptions, domain keywords, and optional reference images.Source: §3.1 · pp. 5–6
  • ROVER-IG scores reasoning process, reasoning visual quality, reasoning-output alignment, visual consistency, and image quality.The five rubric dimensions address both verbal reasoning and generated-image outcomes.Source: §3.1 · p. 6

3.2. Visually-Augmented Reasoning for Verbal Generation

  • ROVER-TG contains 404 tasks across physical world simulation, visual-aided logical problem solving, and enhanced visual perception, using generated images as reasoning intermediates.Tasks include predicting visual states from actions or physical processes, creating visual aids for puzzles and geometry, and supporting challenging perception tasks.Source: §3.2 · p. 6
  • ROVER-TG evaluates intermediate-representation quality, final-answer accuracy, and alignment between generated images and the reasoning outcome.The alignment rubric considers whether visual aids contribute to conclusions and whether image generation was necessary for task completion.Source: §3.2 · p. 7

4. Experiments

The experiments evaluate unified multimodal models on verbally augmented visual generation and visually augmented verbal reasoning, alongside analyses of downstream reasoning and evaluation reliability. Results show that reasoning and alignment track visual generation quality, while visual reasoning artifacts help perceptual tasks more than symbolic ones.

4.2. Verbally-Augmented Reasoning for Visual Generation

  • Reasoning and alignment track ROVER-IG visual generation: open-source models score about 38% lower in reasoning, 31% lower in alignment, and 39% lower in visual generation than closed-source models.The authors describe cross-modal reasoning as a fundamental driver of visual generation effectiveness.Source: §4.2 · p. 9
  • Open-source models with interleaved image-text generation score approximately 38.1% higher than non-interleaved models on the Reasoning Visual metric.The authors suggest that reasoning and generation processes are synergistic in visual expression tasks.Source: §4.2 · p. 9
  • Unified models outperform specialized image-editing models across all visual-quality metrics on ROVER-IG.The paper attributes the editors’ gap on reasoning-dependent tasks to their lack of internal reasoning capabilities.Source: §4.2 · pp. 9–10, Table 4 · p. 11

4.3. Visually-Augmented Reasoning for Verbal Generation

  • The best models reach only 38.8% average interleaved-reasoning quality; visual augmentation helps physical and perception tasks but can hurt logical reasoning.Qwen2.5-VL-7B improves by +3.5% on physical world modeling and +3.8% on visual perception, while VLM logical reasoning degrades by -1.4%.Source: Table 3 · p. 9, §4.3 · p. 10, §4.4 · p. 12

4.4. Further Analyses and Discussions

  • In the cascade comparison, UMM reasoning improves ROVER visual consistency by 20.7%, while GPT-4o refinement lowers both visual consistency and quality on ROVER.The comparison includes FLUX.1 Kontext and its GPT-4o-refined variant; GPT-4o refinement improves FLUX’s EditWorld CLIP-T score by 1.5%.Source: §4.4 · p. 11, Figure 6 · p. 11
  • A controlled study tests whether UMM-generated visual reasoning artifacts improve downstream VLM reasoning across three problem domains.Visual artifacts improve Qwen2.5-VL-7B on physical world modeling and perception, but reduce logical-reasoning performance when the visual reasoning is poor.Source: §4.4 · pp. 11–12, Figure 7 · p. 12
  • Models perform better on temporal, spatial, and causal reasoning than on abstract and mathematical tasks; abstract reasoning correlates weakly with physical reasoning (0.55 to 0.60) but strongly with mathematical reasoning.The authors describe abstract reasoning as a distinct, independent capability from concrete reasoning skills.Source: §4.4 · p. 12
  • GPT-4.1 scores align strongly with human expert judgments across evaluation dimensions, while reasoning-related metrics show larger discrepancies.Figure 9 compares Pearson correlation and mean absolute error for GPT-4.1 against human experts and Gemini-2.5-Pro; the authors report modest evaluator differences and reasonable cross-VLM consistency.Source: §4.4 · p. 12, Figure 9 · p. 13

5. Conclusion

ROVER benchmarks reciprocal cross-modal reasoning in unified multimodal models. Its evaluation of 17 models across 23 task types identifies capability gaps and asymmetries and offers insights for future model development.

5. Conclusion

  • ROVER introduces a benchmark for reciprocal cross-modal reasoning in unified multimodal models.It targets visual generation and interleaved multimodal reasoning.Source: §5 · p. 13
  • Across 23 task types, evaluation of 17 models reveals capability gaps and asymmetries in how current models use cross-modal reasoning.The benchmark includes verbally augmented reasoning for visual generation and visually augmented reasoning for verbal generation.Source: §5 · p. 13
  • ROVER’s analysis identifies factors relevant to reasoning-dependent visual generation, world modeling, and complex reasoning tasks.The authors hope the benchmark informs training paradigms and architectural considerations for future omnimodal models.Source: §5 · p. 13

A. Extended Related Work

Prior work has explored visual representations and interleaved text-image reasoning, while this work examines reasoning across visual and linguistic modalities and their consistency.

A. Extended Related Work

B.1. VLM as Judge

The evaluation uses an automated GPT-4.1 vision-language judge and describes rubric dimensions for judging reasoning and its visual outputs. It presents only temporal and causal variants for two metrics because of space constraints.

B.1. VLM as Judge

  • The Reasoning Process metric assesses verbal reasoning’s logical structure, domain knowledge, type-specific validation, and completeness.Source: §B.1 · p. 21
  • Reasoning Visual scores alignment with target descriptions and correct reasoning principles, while Reasoning Alignment measures consistency between verbal reasoning and generated visuals.Source: §B.1 · p. 21
  • Visual Consistency checks whether non-target elements remain unchanged during reasoning-guided generation.The evaluation uses GPT-4.1 as an automated judge; only temporal and causal variants for RP and RV are demonstrated due to space constraints.Source: §B.1 · p. 21

Unified Models

The section describes unified and image-editing models used in the evaluation, including their architectures, capabilities, and selected inference configurations.

Unified Models

  • Bagel is an open-source multimodal foundation model with 7B active and 14B total parameters, trained on large-scale interleaved multimodal data.It is compared with Qwen2.5-VL and InternVL-2.5 on understanding benchmarks and with Stable Diffusion 3 on text-to-image generation; the experiments use recommended parameters and prompts.Source: Unified Models · p. 21
  • The evaluated unified models include ILLUME+ with dual visual tokenization and diffusion refinement, UniPic2-Metaquery-9B with frozen MLLM and learnable meta-queries, and Ovis-U1 with understanding, generation, and editing capabilities.Their passages specify official inference code or checkpoints for ILLUME+, UniPic2-Metaquery-9B, and Ovis-U1.Source: Unified Models · p. 22
  • Other evaluated models include 37B Emu2-Gen for text-to-image generation and editing, and OmniGen2 with dual-pathway decoding and decoupled image tokenization.OmniGen2 uses a temporal offset of 3.0, text guidance scale 5.0, and image guidance scale 1.5; inference uses a 50-step schedule.Source: Unified Models · p. 22
  • The image-editing baselines comprise six open-source models, including diffusion, visual autoregressive, and MLLM-plus-diffusion approaches.The listed baselines are UltraEdit (SD3), FLUX.1 Kontext, VAREdit-8B, Qwen-Image-Edit, and Step1X-Edit v1.1 and v1.2; the passage says their default hyperparameters were followed.Source: Unified Models · p. 22
  • The reported configurations use 50 inference steps for Qwen-Image-Edit and free-form UltraEdit, and 28 for Step1X-Edit v1.1 and v1.2; v1.2 enables thinking mode and disables reflection.Source: Unified Models · p. 23

B.3. Evaluation Prompt for ROVER-TG

The ROVER-TG prompts ask models to generate intermediate or completed visualizations, inspect them, and use the resulting visual information to answer tasks. They also specify task-specific outputs and constraints for simulation, visual perception, and trajectory planning.

World Model for Embodied Task

  • Trajectory prompts specify ten waypoints as pixel coordinates, requiring a smooth, collision-free path within image boundaries from the current end-effector position to task completion.The prompt also specifies waypoint markers and connecting trajectory lines in the visualization.Source: §B.3 · p. 24

World Model for Physical Task

  • Physical-simulation prompts guide models from analyzing an initial scene and applying physical dynamics to generating and inspecting a final-state image before answering.They mention optional additional scene context, including a second image.Source: §B.3 · p. 25

World Model for Embodied Task

  • Visual-aid prompts require helpful additions to the input, followed by observation of the generated image and reasoning from its visual information.Examples include auxiliary lines, labels, annotations, and constructions.Source: §B.3 · p. 26

Visual Perception for Jigsaw

  • Jigsaw prompts require completing the image region hidden by a gray box, checking consistency with the original, and selecting the matching option.The prompt says not to answer before generating the completed image.Source: §B.3 · p. 27

Visual Perception for Multi-view Reasoning

  • Multi-view prompts ask models to create a wider view containing objects from both images, then infer whether the camera rotated clockwise or counter-clockwise.The generated view is used to reason about the spatial relationship between camera positions.Source: §B.3 · p. 28

B.3. Evaluation Prompt for ROVER-TG

  • Additional templates cover temporal and causal reasoning, reasoning alignment, visual consistency, and image quality.The appendix identifies these as prompt templates for evaluating the corresponding capabilities.Source: Figure 11 · p. 30, Figure 13 · p. 32, Figure 16 · p. 35

Related papers

Loading 2511.01163v1…