PaperlayerView exact revision on arXiv

Source-linked AI summary

TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

Yinyi Luo, Wenwen Wang, Hayes Bai, Hongyu Zhu, Hao Chen, Pan He, Marios Savvides, Sharon Li, Jindong Wang

arXiv:2604.10784v2cs.AI

TL;DR

UMM evaluation lacks a unified solution because results vary across benchmarks and tasks, while architectures, training paradigms, and interfaces remain heterogeneous. TorchUMM addresses this gap with a standardized framework for evaluating, analyzing, and post-training diverse models across understanding, generation, and editing. Controlled evaluations reveal fragmented capabilities, cross-task trade-offs, and backbone-dependent post-training behavior, while tightly coupled perception, reasoning, and generation remain challenging.

  • Problem

    UMM evaluation is fragmented and inconsistent across benchmarks, tasks, architectures, preprocessing pipelines, and model interfaces, limiting reliable cross-model comparison.

  • Method

    TorchUMM standardizes inference, scoring, task pipelines, and post-training across diverse models, benchmarks, and training methods through a unified interface.

  • Results

    Controlled evaluation finds fragmented capabilities, systematic cross-task trade-offs, difficulty with tightly coupled perception-reasoning-generation tasks, and model-dependent post-training effects.

  • Takeaways & Limitations

    TorchUMM enables reliable comparisons and exposes strengths, limitations, and trade-offs that emerge only under controlled, cross-task evaluation.

  • Takeaways & Limitations

    The evaluation is restricted to publicly released checkpoints under standard settings, and alternative prompting or decoding strategies may produce different outcomes.

Abstract

from arXiv · show

Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing across visual and textual modalities. However, developing a unified framework for UMMs remains challenging due to the diversity of model architectures and the heterogeneity of training paradigms and implementation details. In this paper, we present TorchUMM, the first unified codebase for comprehensive evaluation, analysis, and post-training across diverse UMM backbones, tasks, and datasets. TorchUMM supports a broad spectrum of models covering a wide range of scales and design paradigms. Our benchmark encompasses three core task dimensions: multimodal understanding, generation, and editing, and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models and fosters deeper insights into their strengths and limitations, facilitating the development of more capable unified multimodal systems. Code is available at: https://github.com/AIFrontierLab/TorchUMM.

1 Introduction

TorchUMM addresses fragmented and inconsistent UMM evaluation with a unified toolkit for systematic evaluation, analysis, and post-training. Its controlled studies reveal that current models have fragmented capabilities and substantial cross-task trade-offs.

  • Motivation: Existing UMM evaluation is benchmark-dependent, single-task-oriented, and difficult to compare across models because pipelines, preprocessing, and interfaces differ.These conditions can produce overly optimistic conclusions about model capability.
  • TorchUMM: TorchUMM standardizes evaluation pipelines across heterogeneous models and tasks for controlled, reproducible comparisons.It abstracts model-specific details through a standardized interface and supports flexible post-training integration.
  • TorchUMM: TorchUMM covers multimodal understanding, generation, and editing while providing modular support for diverse post-training methods.The framework is designed to offer a holistic view across multimodal capabilities.
  • Findings: No single UMM dominates understanding, generation, and editing, while strong performance in one dimension often comes at the expense of others.The evaluation also finds that model scale alone is not a reliable performance predictor and that architecture and training strategies matter.
  • Findings: Post-training methods often introduce non-trivial performance trade-offs across tasks and benchmarks rather than holistic improvements.These findings underscore the importance of evaluating post-training methods across multiple tasks.

2 Related Work

UMM research spans multiple architectural paradigms, while its evaluation ecosystem remains fragmented across understanding, generation, editing, and cross-task benchmarks.

  • Unified Multimodal Models: UMMs commonly use decoder-only autoregressive transformers, hybrid autoregressive-diffusion or flow-based frameworks, and modular or lightweight designs.These paradigms target multimodal understanding, generation, and cross-modal alignment in different ways.
  • Related Benchmarks and Codebase: Understanding benchmarks assess reasoning, perception, and knowledge integration, whereas generation benchmarks measure fidelity, prompt alignment, and compositionality.Editing benchmarks focus on instruction-based image modification, and UMM-specific benchmarks target cross-task generalization.

3 TorchUMM

TorchUMM provides a modular, standardized codebase spanning heterogeneous UMMs, three core tasks, benchmark datasets, and unified post-training methods. Its design supports direct comparisons and extension across models, datasets, and training approaches.

  • Architecture: TorchUMM uses four layers covering infrastructure, core functionality, task execution, and applications or APIs.The layers provide extensibility, model and dataset support, unified task pipelines, configuration interfaces, and evaluation reports.
  • Architecture: The codebase currently supports 15 models, 12 benchmark datasets, and 5 post-training methods.Its modular design makes it easy to extend support for additional models, datasets, and training methods.
  • Model Support: TorchUMM abstracts model-specific implementations into a unified interface for direct and fair comparisons across heterogeneous architectures.Supported backbones span different design paradigms and scales, with only a subset natively supporting image editing.
  • Core Tasks: Multimodal understanding evaluates perception, visual question answering, and reasoning over visual inputs.The task includes both low-level perception and high-level cognitive understanding.
  • Core Tasks: Multimodal generation assesses text-to-image synthesis through compositionality, semantic alignment, and world-knowledge incorporation.The task focuses on producing images aligned with textual prompts.
  • Core Tasks: Multimodal editing evaluates instruction-based modification of existing images, including fine-grained control, consistency preservation, and instruction-following.Editing measures how models manipulate images according to textual instructions.
  • Post-Training: TorchUMM unifies post-training evaluation across SFT, IRG, UniCoT, RecA, and UniGame despite differences in pipelines, implementation details, and datasets.This unified interface enables fair and reproducible comparisons among post-training approaches.

4 Evaluation Results

TorchUMM evaluates heterogeneous unified multimodal models under a controlled protocol across generation, understanding, editing, and unification tasks. Results reveal fragmented capabilities, architecture- and training-dependent performance, and substantial cross-task trade-offs.

  • Identical preprocessing, inference settings, and scoring pipelines enable direct cross-architecture comparisons across generation, understanding, and editing tasks.All supported models were independently reproduced under TorchUMM’s unified protocol.
  • 4.1 Generation: No single model dominates text-to-image generation: Ovis-U1 leads GenEval at 90.05, DeepGen leads DPG Bench at 87.44, and Emu3.5 leads WISE at 0.633.Ovis-U1 also achieves the strongest UEval result at 34.17, while the WISE spread from Emu3.5 to Janus is 0.633 to 0.222.
  • 4.2 Understanding: Bagel leads the major multimodal understanding metrics, while a persistent perception–cognition gap and generation–understanding trade-off remain across models.Bagel reaches MMMU 0.519 and MM-Vet 65.9; Emu3.5, despite strong generation, has the lowest MME perception score among models with understanding capability at 832.17.
  • 4.3 Image Editing: Emu3.5 is the strongest overall editing model, but SC–PQ disparities and stronger multi-turn differences show that semantic correctness and perceptual quality are only partially aligned.Emu3.5 scores 7.56 on both GEdit-EN and GEdit-CN, with ImgEdit scores of 4.24 SingleTurn, 4.89 MultiTurn, and 4.88 UGE.
  • 4.4 Unification Evaluation: Unification remains fragile: performance drops with structure-aware visual encoders, compositional and multi-step reasoning is limited, and scale alone does not guarantee better results.Janus declines from 0.8465 with CLIP to 0.5691 with DINOv3, while 34B Emu3.5 reaches only 0.5093 in the cited unification evaluation.

5 Analysis

TorchUMM’s architectural analysis finds that greater parameter and token sharing does not necessarily produce stronger representational unification. Across instruction following, reasoning traces, and latent alignment, OmniGen2 often outperforms the more aggressively unified MMaDA and Show-o2.

  • Architectural unification: Architectural unification is not equivalent to representational unification across MMaDA, Show-o2, and OmniGen2.Their architectural unification follows MMaDA > Show-o2 > OmniGen2, but several capability and representation orderings reverse that prior.
  • Instruction following: OmniGen2 shows the strongest instruction-following behavior on UEval despite being the least architecturally unified model.It better preserves procedural structure in step-by-step drawing prompts and more often produces recognizable technical layouts for diagrams.
  • Reasoning traces: On MathVista, OmniGen2 most reliably follows chain-of-thought instructions, produces long explicit traces, and achieves the best open-source accuracy.MMaDA and Show-o2 apply more aggressive sharing across modalities and tasks but show weaker reasoning-trace behavior in the cited comparison.
  • Cross-modal alignment: OmniGen2 has the largest CMAM, 0.126, versus 0.030 for Show-o2 and 0.019 for MMaDA, indicating tighter paired text–image alignment.Larger CMAM corresponds to a smaller effective modality gap.

6 Conclusion and Discussion

TorchUMM standardizes evaluation, analysis, and post-training across diverse UMMs, revealing capability gaps and trade-offs that fragmented protocols can miss. Its conclusions are bounded by publicly released checkpoints, standard settings, and incomplete coverage of modalities and design choices.

  • Contribution: TorchUMM provides a unified framework for evaluating, analyzing, and post-training UMMs across diverse models, benchmarks, and training methods.It standardizes inference and scoring under a shared pipeline for reliable comparisons.
  • Findings: Controlled evaluation reveals a perception–cognition gap, a semantic-correctness–perceptual-quality mismatch, and difficulty with iterative structured visual manipulation.These patterns emerge when understanding and generation are evaluated jointly.
  • Findings: Post-training methods behave inconsistently across backbones, sometimes improving one model while degrading another.The analysis therefore identifies strong backbone dependence in post-training outcomes.
  • Limitations: The evaluation is restricted to publicly released checkpoints under standard settings, so alternative prompting or decoding may produce different outcomes.This is an explicit scope boundary of the reported comparisons.
  • Limitations: The benchmark suite focuses largely on English and static images and samples only part of the broader design space across models, methods, and data.Video, audio, human preference modeling, and additional design combinations remain outside the current coverage.

A.1 Backbone Adapter Architecture

TorchUMM uses a standardized backbone-adapter interface and YAML-driven pipelines to separate model-specific inference from evaluation and post-training. This minimal-touch design lets researchers switch models, benchmarks, and training methods without modifying core pipeline code.

  • Backbone adapters: The BackboneAdapter protocol defines the minimal interface that every supported model must implement.The interface includes model naming, loading, and generation operations.
  • Backbone adapters: Self-contained adapter modules hide device mapping, tokenization, attention implementation, and output formatting behind the standardized interface.This isolates model-specific concerns from shared evaluation and inference logic.
  • Configuration and stages: TorchUMM organizes behavior into inference, evaluation, and post-training stages driven entirely by YAML configurations.Configurations specify model settings, benchmark bindings, scoring choices, training methods, optimizer settings, checkpoint schedules, and distributed parameters.
  • Configuration and stages: Changing models on the same benchmark requires only updating the backbone name and model path, with no code modifications.Each evaluation run is fully determined by its configuration file, supporting reproducible experimentation.
  • Evaluation: Evaluation runners normalize benchmark inputs through a shared InferencePipeline and save structured outputs for analysis or external scoring.Two-stage benchmarks can call official scripts through thin wrappers while preserving the common pipeline.
  • Extensibility: New models, benchmarks, and post-training methods follow a minimal-touch extension pattern using adapters, handlers, configuration files, and registered entry points.Trained checkpoints can be evaluated through existing configurations by changing the model weights path.

B.1 Supported Datasets and Benchmarks

TorchUMM evaluates UMMs across understanding, generation, editing, and unified cross-task settings. Its benchmark suite combines task-specific datasets with integrated protocols and supports both direct evaluation and post-training followed by evaluation.

  • Generation benchmarks: Generation benchmarks assess detail preservation, prompt fidelity, compositional generalization, and related text-to-image capabilities.The cited suite includes DPG-Bench and GenEval among representative generation evaluations.
  • Understanding benchmarks: Understanding benchmarks evaluate expert reasoning, multimodal perception, knowledge integration, and instruction following.MMMU, MMBench, and MME provide complementary assessments of these abilities.
  • Unified and cross-task benchmarks: Unified and cross-task benchmarks jointly assess multiple multimodal capabilities within integrated evaluation settings.UEval spans generation and reasoning scenarios, while Uni-MMMU jointly evaluates understanding, generation, and editing.
  • Editing benchmarks: Editing benchmarks measure precise attribute manipulation, multi-step edits, instruction alignment, consistency preservation, and visual coherence.GEdit-Bench and ImgEdit target complementary aspects of instruction-based image modification.
  • Evaluation protocols: TorchUMM supports both single-stage inference-and-scoring evaluation and multi-stage post-training followed by evaluation.This enables flexible assessment protocols across the benchmark suite.

C Detailed Results

TorchUMM reports fine-grained subscores for generation, understanding, and editing across diverse benchmarks, enabling more detailed performance analysis.

  • C Detailed Results: Fine-grained subscores cover generation, understanding, and editing tasks across the unified evaluation framework.The reported benchmarks include GenEval, WISE, MathVista, MMMU, GEdit-EN, RealUnify, and uni-MMMU.

D.1 Unification Degree is Not a Reliable Capability Proxy

The UEval study finds that greater architectural unification is not a reliable proxy for capability: the least unified model performs best on the examined coupled tasks. However, the comparison is confounded by differences in backbones, generative formulations, and training data.

  • D.1 Unification Degree is Not a Reliable Capability Proxy: OmniGen2, the least unified system, outperforms Show-o2 and MMaDA on both representative UEval cases.The models are ordered by architectural unification as MMaDA > Show-o2 > OmniGen2.
  • D.1 Unification Degree is Not a Reliable Capability Proxy: 0.79 vs. 0.46 and 0.29: OmniGen2 achieves the highest rubric-based score on the step-by-step cartoon-dog tutorial.The task requires ordered visual progression aligned with textual explanations.
  • D.1 Unification Degree is Not a Reliable Capability Proxy: 0.13 vs. 0.07 and 0.00: OmniGen2 produces the only output resembling the requested structured Transformer diagram.All three models struggle with the technical diagram task.
  • D.1 Unification Degree is Not a Reliable Capability Proxy: The capability ordering is nearly the reverse of the architectural prior, with OmniGen2 outperforming the more unified models in the comparison.The authors caution that this is not a clean causal estimate of unification itself.
  • D.1 Unification Degree is Not a Reliable Capability Proxy: The comparison is confounded by differences in backbone inheritance, generative formulation, and uncontrolled training-data composition and quality.The authors state that the practical benefits of unification remain entangled with these factors.

D.2 Heterogeneous Effects of Unification Training

Unification training has heterogeneous effects: a relatively modular recipe preserves backbone-like behavior, whereas stronger sharing reshapes consistency and reasoning behavior more aggressively.

  • D.2 Heterogeneous Effects of Unification Training: OmniGen2 remains behaviorally close to its Qwen2.5-VL-3B-Instruct backbone under the lower-unification pairing.Its response-consistency curves and layerwise latent trajectories nearly overlap with the backbone’s.
  • D.2 Heterogeneous Effects of Unification Training: Show-o2 exhibits broader output variation and altered internal dynamics than its Qwen2.5-VL-7B-Instruct reference under the higher-unification pairing.The reference retains a tight, high-stability profile, while Show-o2 spreads into the low-similarity region.
  • D.2 Heterogeneous Effects of Unification Training: Stronger unification training can reshape inherited backbone behavior more aggressively, but stronger change is not automatically better.The comparison contrasts relatively modular and more strongly shared recipes.
  • D.3 Reasoning Traces Expose a Test-Time Scaling Gap: 99.5%: OmniGen2 produces explicit CoT traces for almost all samples, with a median visible length of 220 tokens and 58.1% accuracy.Its entropy decreases gradually over the trace, suggesting progressively constrained answers.
  • D.3 Reasoning Traces Expose a Test-Time Scaling Gap: MMaDA and Show-o2 produce median traces of 2 and 5 tokens, with accuracies of 29.3% and 50.0%, respectively.They emit answer-only responses on 63.1% and 49.5% of samples, respectively.
  • D.3 Reasoning Traces Expose a Test-Time Scaling Gap: The more aggressively unified models often fail to preserve or activate backbone CoT-following behavior, creating a trace-formation gap.Additional test-time reasoning instructions therefore do not yield the visible reasoning process rewarded by MathVista.

D.4 Unification Does Not Necessarily Reduce the Modality Gap

TorchUMM examines whether architectural unification produces representational alignment and finds that it does not necessarily reduce the effective modality gap. Controlled comparisons are needed because the observed effects remain entangled with model and training differences.

  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: CMAM compares each sample’s paired text–image representation with mismatched text–image pairs to quantify shared latent-space coherence.Higher CMAM indicates stronger paired alignment and a smaller modality gap.
  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: 0.126: OmniGen2 has the largest final-layer CMAM despite being the least unified system by design.Its paired text and image spans visually interleave in the final-layer manifold.
  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: 0.030 and 0.019: Show-o2 and MMaDA have much smaller final-layer CMAM values despite greater architectural unification.Their paired representations are less distinguishable from mismatched pairs.
  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: The representational results align with OmniGen2’s stronger UEval instruction following, backbone-like consistency, and CoT-following behavior.The authors present the smaller effective modality gap as a unifying explanation for these earlier behavioral findings.
  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: Architectural unification is not equivalent to representational unification, and aggressive parameter sharing can increase the practical modality gap before sufficient scale, data, or optimization.The authors call for controlled studies varying only the unification mechanism while fixing backbone, tokenizer, data mixture, and optimization recipe.
  • D.4 Unification Does Not Necessarily Reduce the Modality Gap: TorchUMM relies on existing datasets and models, so evaluations may inherit their biases and incomplete coverage of real-world scenarios.The paper recommends interpreting results cautiously and complementing them with additional analysis in practice.
Loading 2604.10784v2…