PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models

Rohit Patel, Dieuwke Hupkes, Sloan Strader

arXiv:2608.26317v1cs.CVcs.AIcs.MM

TL;DR

Existing evaluations rarely test models across multiple non-text inputs and outputs. The paper introduces MMI, a five-modality benchmark with modality-specific rubrics and a separate presence score, finding low modality-return performance across five frontier models and 70.8% judge–human agreement in a tool-enabled validation experiment.

  • Problem

    Existing evaluation frameworks are predominantly bimodal, while benchmarks requiring multiple non-text modalities in both input and output are virtually nonexistent.

  • Method

    MMI uses 893 self-contained prompts across text, image, audio, video and document, with up to three input and output modalities, human-authored rubrics, MMI Value, and MPS.

  • Results

    MPS ranges from 15.6 for Claude Opus 4.6 to 34.9 for GPT-5.4, while rubric-based LLM judging agrees with human annotators on 70.8% of judgments.

  • Takeaways & Limitations

    Current LLMs remain predominantly text-output in practice, and MMI is designed to track progress toward broader multimodal responses.

  • Takeaways & Limitations

    The main MMI Value is not well tested because evaluated models return too few assets, and rubric–human agreement is imperfect.

Abstract

from arXiv · show

Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model's MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments.

1 Introduction

Existing evaluations rarely test multimodal systems across multiple non-text inputs and outputs, motivating MMI, a benchmark with modality-specific scoring and a separate presence metric. Across five frontier models, returned modalities remain scarce, so MPS is reported as the main result while rubric-based judging is evaluated separately.

  • Motivation: Current release benchmarks largely omit modalities and report no cross-modal output evaluations.Existing multimodal benchmarks are predominantly bimodal, while fully cross-modal processing is virtually absent.
  • Benchmark: MMI contains 893 self-contained prompts spanning five modalities, with up to three input and output modalities and human-authored rubrics.The modalities are text, image, audio, video and document.
  • Scoring: MMI Value averages rubric scores equally across expected output modalities, while MPS measures modality production using per-prompt F1.The separate metrics distinguish incorrect content from failure to return an expected modality.
  • Results: 15.6–34.9: MPS spans only 15.6 for Claude Opus 4.6 to 34.9 for GPT-5.4 across five evaluated models.Because models returned too few assets for meaningful MMI Value scores, MPS is the main reported result.
  • Rubric validation: 70.8%: rubric-based LLM judging agreed with human annotators on judgments from a separate tool-enabled experiment.The experiment produced gradeable outputs by giving three models tools to generate images, videos and audio.

2 Related work

Prior multimodal evaluation is organized mainly around individual modality pairs or isolated generation tasks, with limited coverage of jointly chosen cross-modal outputs. MMI extends this landscape across five input and output modalities while evaluating both modality selection and returned-content quality.

  • Bimodal understanding: Vision-language benchmarks mostly pair visual inputs with textual questions and answers, despite broad perception and reasoning coverage.Video-text benchmarks extend this setup to temporal reasoning over videos.
  • Audio and documents: Audio-language benchmarks cover speech, music and environmental sound understanding, while document benchmarks remain text-output tasks over document inputs.These benchmarks target narrower modality-specific settings rather than jointly composed outputs.
  • Generation: Multimodal generation benchmarks mostly isolate image output and presuppose a generator instead of testing whether a model chooses a relevant output modality.MMI directly evaluates that output-modality choice.
  • Cross-modal benchmarks: Existing cross- or any-to-any benchmarks combine multiple modalities but generally produce text only, separate understanding from generation, or do not test cross-modal output decisions.This limits their coverage of jointly selected non-text outputs.
  • MMI positioning: MMI spans five modalities in both inputs and outputs, composes up to three modalities per prompt, and scores modality choice alongside content quality.Human-authored modality-specific rubrics support content assessment; MPS isolates production.
  • Comparison framework: Table 1 records whether academic benchmarks support text, image, audio, video and document inputs and outputs, plus cross-modal output choice.Its symbols distinguish support, non-support and partial support.

3 Modality Maturity Index: the benchmark

MMI is an 893-prompt benchmark spanning five modalities and combinations of up to three input and output modalities. It combines modality-specific correctness rubrics with MMI Value and MPS measures to distinguish response quality from modality presence.

  • MMI prompts: 893 prompts cover text, audio, image, video and document modalities, with examples spanning single and multiple inputs and outputs.The benchmark includes 475 one-input/one-output prompts, 264 multiple-input/single-output prompts, 76 single-input/multiple-output prompts, and 78 multiple-input/multiple-output prompts.
  • MMI prompts: MMI’s five modalities are more evenly represented in outputs than inputs, where text is especially overrepresented.Text commonly accompanies other modalities for clarification or explanation, and most non-text or non-audio prompts include text or audio alongside them.
  • Data collection: The dataset was developed through piloting, manual review, iterative refinement, and staged expansion from text-input prompts to interleaved input and output modalities.The initial pilot contained 50 questions, after which later stages added interleaved modalities in inputs and in both inputs and outputs.
  • Rubrics: Human-authored rubrics assess whether each returned output modality contains correct content, rather than merely whether that modality appears.Rubrics are designed to judge each output modality in isolation without requiring external knowledge or access to other returned modalities.
  • Rubrics: The rubric framework does not attempt joint correctness judgments for combinations such as synchronized generated audio and video.Such judgments would require processing more modality combinations than answering the benchmark prompts themselves.
  • Scoring: MMI Value macro-averages rubric scores across expected output modalities, while MPS macro-averages modality-set F1 and ignores output-content correctness.Absent desired modalities receive zero MMI Value, motivating MPS as a diagnostic for missing modalities; complementary metrics include precision, recall, pass rate, and input failure rate.

4 Experimental setup

The experiments apply MMI to five frontier multimodal models under the normal protocol and separately use generation tools to validate the rubrics. The setup measures modality support, returned-modality presence, and rubric-based correctness, while keeping tool-assisted validation separate from model comparison.

  • Main evaluation: The main evaluation applies MMI to five models: GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, Gemini 3.1 Flash, and Llama 4 Maverick.The first four models were accessed through provider APIs, while Llama 4 Maverick was served locally from an open-weight checkpoint.
  • Presence detection: Modality presence is detected through native returned assets, links to modality-specific hosting platforms, or fallback judge detection.Strict presence counts only native assets, whereas lenient presence counts any of the three detection routes; reported presence values are generally lenient.
  • Rubric validation: The tool-scaffolding experiment gives three models image, video, and audio generation tools to produce enough assets for rubric validation, not to rank models.Its numbers are not comparable with the main evaluation because the experiments have different purposes and settings.
  • Output-modality metrics: Figure 3 reports F1, recall, precision, and native precision per output modality, averaging across the dataset rather than per-prompt MPS scores.Precision is set to NaN when a model never returns a modality, and those cells are excluded from averages.
  • Output-modality results: The main evaluation’s low asset availability limits modality-difficulty comparisons: GPT-5.4 is the only model natively generating a non-text modality, while no model generates audio or audio links.Consequently, per-modality metrics mainly reflect whether models return a modality at all rather than its intrinsic difficulty.
  • Rubric validation: LLM judges grade each output modality against human-written rubrics, while human annotators judge outputs directly without seeing those rubrics.Agreement between these judgments is used to assess whether rubric-based LLM judging can approximate MMI Value.

5 Results

Across five models, modality presence is low, with recall and pass rates revealing frequent omissions and input failures. Rubric validation is possible on tool-generated assets, but documents remain unvalidated because no document-generation tool was provided.

  • 5.1 Overall presence scores: MPS, precision and recall are reported for all models, with precision near 100 but recall no higher than 33.8.The reported recall scores are 33.8 for GPT-5.4, 25.4 for Gemini 3.1 Flash, 23.6 for Gemini 3.1 Pro, 15.1 for Llama 4 Maverick and 14.5 for Claude Opus 4.6.
  • 5.2 Input failure rate and pass rate: Input failure rates range from 8.0% for Gemini 3.1 Pro to 37.0% for Claude Opus 4.6, separating input acceptance failures from wrong output modalities.Reading input failure rate alongside recall distinguishes these two deficiencies.
  • 5.2 Input failure rate and pass rate: Pass rate falls from 31.0 for GPT-5.4 to 11.3 for Claude Opus 4.6, showing that partial fulfillment and omitted modalities are common.Pass rate asks whether every expected output modality appears, unlike recall, which averages modality-level success.
  • 5.3 The impact of output modality: Per-modality metrics are difficult to interpret because few models natively support non-text outputs; no model generates audio, and only GPT-5.4 natively generates another modality.The authors therefore conclude that non-text modalities leave substantial room for improvement.
  • 5.4 Rubric validation results: Tool-scaffolded models produce outputs for 71% of modalities, or 76% excluding documents, compared with about a quarter in the main evaluation.This second experiment provides enough generated assets to validate the rubrics, while document outputs remain unavailable.

6 Conclusion

MMI reveals that current multimodal models remain predominantly text-output systems, while the benchmark’s main correctness index is not yet meaningfully evaluated because models return too few assets.

  • MMI evaluates 893 prompts spanning up to three of five modalities in both inputs and outputs, with diverse topics and modality combinations.
  • 15.6–34.9 MPS across evaluated models indicates that returning the correct output modalities remains challenging even before answer correctness is considered.
  • Audio is the hardest output modality: none of the evaluated models supports or returns it, while image and video remain difficult despite higher scores.
  • 70.8% rubric-blind judge–human agreement supports rubric validation, although disagreement reflects judge errors, open-ended correctness, and single-modality versus unaided judgments.
  • MMI’s main index is not meaningfully tested because the models return too few assets, and rubric agreement remains imperfect.
  • The reported scores characterize bare provider APIs rather than deployed assistants, which may add image, audio, or video generators.

Appendix

The appendix specifies separate judges for detecting genuinely produced modalities and for scoring rubric criteria against isolated modality payloads.

  • The modality-presence judge examines a model’s API response to identify which of Text, Image, Audio, Video, and Document were genuinely produced.
  • Text requires substantive output, while image, audio, and video require native data, encoded content, or generated-media URLs.
  • Descriptions, screenplays, lyrics, and refusals do not count as production of the corresponding non-text modality.
  • The presence judge returns JSON containing detected_modalities and a brief reasoning field.
  • The rubric judge receives the original prompt, expected modalities, input metadata, one rubric criterion, and only the matching model-produced payload.
  • The rubric judge returns a JSON score from 0.0 to 1.0, using partial credit when a criterion is partly satisfied.
  • Non-text artifacts are attached with labels separating generated outputs from prompt inputs, which are context only.

C Full modality heatmap

Figure 6 presents the distribution of input and output modalities across the full MMI benchmark.

  • Figure 6’s full modality grid shows how MMI prompts distribute input and output modalities.

Guiding principle

MMI rubrics are designed as isolated, objective grading instruments that specify observable requirements for each expected output modality.

  • A rubric captures only properties whose presence or absence settles correctness, rather than fully specifying an asset.
  • Each criterion is graded from one output modality alone and cannot depend on other outputs, prompts, or input assets.
  • Criteria must state the required fact, value, or property explicitly so graders do not derive answers or rely on outside knowledge.
  • For edits or transformations, criteria describe concrete output properties rather than requiring comparison with the input.
  • Rubric construction confirms modalities first, then prioritizes integrity and content criteria, with limited craft criteria when execution affects usefulness.
  • Criteria should be self-contained, atomic, mutually exclusive, declarative, and binary-decidable.
  • Open-ended prompts should be judged through observable fitness-for-purpose properties rather than subjective impressions such as visual appeal.
  • Rubrics target 4–12 criteria per output modality, with equal within-modality weighting and per-modality averaging before prompt-level averaging.

Worked example

The worked example shows how a prompt names an output file type, structure, and acceptable content criteria across modalities. A crochet rubric illustrates the same multi-modality design, with each modality answering a different prompt clause.

  • Worked example: The example requests a CSV document and specifies file type, tabular structure, and acceptable content requirements.The rubric checks a delivered CSV, valid headers and comma-separated rows, and multiple pros and cons for each car type.
  • Worked example: The crochet rubric assigns different prompt clauses to different modalities, including pronunciation in the audio criterion.The audio criterion carries the expected pronunciation instead of deferring to the video.

E Human annotation guidelines

Human annotators assess whether model answers correctly satisfy each desired output modality in multimodal questions. They use binary modality-level judgments with comments and apply explicit rules for missing, embedded, extra, linked, and misaligned outputs.

  • E Human annotation guidelines: Annotators receive a multimodal question, its desired output modalities, and the model-generated answer.The benchmark covers text, audio, image, video, and document outputs.
  • E Human annotation guidelines: Each desired output modality receives a binary correct-or-incorrect assessment and a short justification.Annotators judge every required modality, including modalities the model failed to return.
  • E Human annotation guidelines: Links count as returned assets unless broken, while correctly produced but misaligned modality pairs may receive one incorrect judgment.These rules cover linked assets and questions requiring alignment between modalities such as audio and video.
  • E Human annotation guidelines: Nested modalities are not scored separately, and accompanying or unrequested text is judged only when text is itself required.A PDF counts as document rather than separate text or image, and audio inside a requested video is not separately assessed.
  • E Human annotation guidelines: Annotators may use AI to acquire expert knowledge but may not submit the question or model answer to an AI for correctness judgments.The guidance prohibits asking an AI whether the answer is correct or using an AI response to judge it.

Example

The cookie example demonstrates per-modality assessment against three requested outputs: text, video, and image. The answer satisfies the video requirement but fails the text and image requirements for different reasons.

  • Example: The prompt requests a cookie ingredient list, tutorial video, and invitation image.Its desired output modalities are text, video, and image.
  • Example: The model answer provides a video tutorial link and lists only some ingredients.The displayed answer includes a tutorial URL and two ingredient lines.
  • Example: Video is correct because the model returned a tutorial, while text is incorrect because baking soda and chocolate chips are missing.The text assessment requires the ingredient list to include those omitted items.
  • Example: Image is incorrect because the model did not return an image.The missing image is scored as an incorrect required modality.

F MMI Values for the rubric-validation runs

The rubric-validation experiment documents MMI Values for three tool-scaffolded models rather than ranking the frontier models. Table 5 defines the aggregate and document-excluded scores used for this separate run.

  • F MMI Values for the rubric-validation runs: The Section 4.3 experiment exercises the rubrics and does not rank models or provide values comparable to Table 4.It uses a later set of models with Gemini generation backends and documents the numbers underlying Section 5.4.
  • F MMI Values for the rubric-validation runs: Table 5 reports MMI Values for three tool-scaffolded models, including all criteria and a document-excluded variant.The all-modality score averages prompt-level output-modality criterion means across 893 prompts; the document-excluded score uses prompts retaining non-document outputs.
Loading 2608.26317v1…