PaperlayerView exact revision on arXiv

Source-linked AI summary

JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction

Atsuyuki Miyai, Shota Onohara, Jeonghun Baek, Kiyoharu Aizawa

arXiv:2512.14620v1cs.CLcs.AIcs.CV

TL;DR

Japanese LMM evaluation lacks benchmarks for tasks that integrate image-embedded text and visual information. JMMMU-Pro puts each question’s text and image into one image and uses a generation-led construction process; experiments find open-source LMMs struggle substantially, with the best-performing open-source model below 50%.

  • Problem

    Japanese LMMs lack evaluation on tasks requiring integrated visual-textual understanding through visual perception, despite the limited Japanese evaluation benchmarks.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    JMMMU-Pro embeds JMMMU question text and images in a single image, while Vibe Benchmark Construction uses an image-generation model to produce questions that humans verify and refine when needed.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    All open-source LMMs struggle substantially on JMMMU-Pro, with the best-performing open-source model below 50% and a substantial gap from strong reasoning-based closed-source models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    JMMMU-Pro offers a more rigorous evaluation of Japanese multimodal capabilities, and Vibe Benchmark Construction is presented as a framework for scalable future image-based VQA benchmark development.

    Source: §6 · p. 10

    The OCR analysis is scoped to correlations between OCR performance and JMMMU-Pro accuracy across several LMMs.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

This paper introduces JMMMU-Pro, an image-based Japanese Multi-discipline Multimodal Understanding Benchmark, and Vibe Benchmark Construction, a scalable construction method. Following the evolution from MMMU to MMMU-Pro, JMMMU-Pro extends JMMMU by composing the question image and question text into a single image, thereby creating a benchmark that requires integrated visual-textual understanding through visual perception. To build JMMMU-Pro, we propose Vibe Benchmark Construction, a methodology in which an image generative model (e.g., Nano Banana Pro) produces candidate visual questions, and humans verify the outputs and, when necessary, regenerate with adjusted prompts to ensure quality. By leveraging Nano Banana Pro's highly realistic image generation capabilities and its ability to embed clean Japanese text, we construct a high-quality benchmark at low cost, covering a wide range of background and layout designs. Experimental results show that all open-source LMMs struggle substantially with JMMMU-Pro, underscoring JMMMU-Pro as an important benchmark for guiding future efforts in the open-source community. We believe that JMMMU-Pro provides a more rigorous evaluation tool for assessing the Japanese capabilities of LMMs and that our Vibe Benchmark Construction also offers an efficient guideline for future development of image-based VQA benchmarks.

1 Introduction

JMMMU-Pro addresses the lack of Japanese evaluation for tasks that integrate question text and images through visual perception. The paper introduces a single-image benchmark and a generation-led construction method, then reports substantial difficulty for open-source LMMs.

1 Introduction

  • JMMMU provides a foundation for comparison through 720 culture-agnostic items translated from MMMU and 600 culture-specific items incorporating Japanese cultural elements.The culture-agnostic subset supports comparison with MMMU, while the culture-specific subset evaluates cultural understanding.Source: §1 · p. 2
  • Existing Japanese benchmarks present question images and question text as separate modalities, while no Japanese benchmark supports the unified visual evaluation described here.The paper motivates this setting by noting that users commonly provide screenshots containing both Japanese text and images.Source: §1 · p. 2
  • JMMMU-Pro embeds each of JMMMU’s 1,320 question texts and images into a composite image, enabling comparison with JMMMU as a signal of visual cognitive ability.The benchmark targets integrated visual-textual understanding through visual perception.Source: §1 · pp. 2–3
  • Vibe Benchmark Construction has an image-generation model produce visual questions, with humans checking outputs and adjusting prompts before regeneration when needed.Nano Banana Pro generates realistic images and can embed Japanese text, producing varied backgrounds and layouts.Source: §1 · pp. 2–3
  • Vibe Benchmark Construction is described as scalable, requiring minimal human effort, and enabling controllable generation of diverse layouts.Approximately 95% of JMMMU-Pro questions were generated by Nano Banana Pro and subsequently checked by humans.Source: §1 · p. 3
  • The evaluation covers 14 LMMs, including closed-source, English-centric open-source, multilingual open-source, and Japanese open-source models.The reported findings include near-random performance for many LMMs and a substantial gap between closed-source and open-source models.Source: §1 · p. 3
  • The best-performing open-source model scores below 50% on JMMMU-Pro, and open-source models show performance decreases of up to 23% versus JMMMU.Strong reasoning-based closed-source models perform considerably well, while strong OCR alone is insufficient to solve the benchmark.Source: §1 · p. 3

2 Related Work

Prior benchmarks and models leave Japanese integrated visual-textual reasoning under-evaluated. The related work positions JMMMU-Pro alongside English benchmarks and image-generation approaches while identifying gaps in Japanese coverage and scalable question construction.

Related Work

  • Japanese LMMs have not been evaluated on tasks requiring Japanese questions to be solved through integrated visual-textual understanding.The paper presents this as a need for a dedicated Japanese benchmark.Source: §2 · p. 3
  • MMMU-Pro combines question text and images in one image, but evaluates only English, leaving other languages unassessed.Unlike OCR benchmarks or DocVQA, it also requires reasoning that integrates recognized text and visual information.Source: §2 · p. 4
  • Japanese benchmarks either lack expert-level coverage or do not combine image-embedded text with the complex reasoning demanded by MMMU-Pro.JMMMU advances Japanese evaluation, while JDocQA [32] and MangaVQA [5] include Japanese text in images but are described as insufficient for that reasoning level.Source: §2 · p. 4
  • Prior image-generation approaches still require substantial effort to create question text, whereas Vibe Benchmark Construction makes image models the primary driver of VQA-image generation.For LogicOCR [55], the paper notes reduced photorealism and dataset shrinkage from filtering; its proposed pipeline uses human verification and prompt adjustment when necessary.Source: §2 · p. 4

3 JMMMU-Pro Benchmark

JMMMU-Pro contains the original JMMMU questions as composite images, and the paper describes how the benchmark is constructed and checked. Vibe Benchmark Construction delegates image-based VQA generation to an image model, with humans verifying outputs and refining prompts when needed.

3 JMMMU-Pro Benchmark

  • JMMMU-Pro retains all 1,320 JMMMU questions and their contents while presenting them as image-based questions.The construction pipeline is illustrated in Figure 2.Source: §3 · p. 4

3.1 Revisiting the JMMMU Benchmark

  • JMMMU has 1,320 questions across 28 subjects, including 720 culture-agnostic and 600 culture-specific questions; the authors converted 50 open-ended items and revised two samples.The revised version is JMMMU-verified-2025-12, and all JMMMU scores in the paper use it.Source: §3.1 · pp. 4–5

3.2 Definition of Vibe Benchmark Construction

  • Vibe Benchmark Construction has an image-generation model produce VQA problem images, while humans verify outputs and adjust prompts as needed.Unlike earlier approaches described here, the image-generation model carries out the VQA creation process rather than producing only the visual component.Source: §3.2 · p. 5

3.3 Detailed Pipeline of Vibe Benchmark Construction

  • Image generation varies backgrounds, background colors, fonts, margins, capture states, and aspect ratios to create diverse benchmark images.Options include nine backgrounds, six background colors, five font styles, two margin sizes, three capture states, and four aspect ratios; Figure 3 reports background and color statistics, and Figure 4 shows samples.Source: Figure 3 · p. 5, §3.3 · p. 6, Figure 4 · p. 6
  • The prompt omits special image-tag instructions because the authors found they significantly degraded generated-image quality.The authors hypothesize that such instructions may interfere with the model's generation process.Source: §3.3 · p. 6
  • Reviewers checked generated text and images against originals, allowed valid image-tag variations, and regenerated failed items; 71% passed the first review.Failures included unrelated replacement images, unreadable text, missing or incorrect question text, and visually unnatural images; a final crosscheck followed several review rounds.Source: §3.3 · pp. 6–7
  • Authors manually created 67 cases the model struggled to generate: 16 with long text, 36 with small or difficult text, 2 with extreme aspect ratios, 8 with formulas or notation, and 5 rejected by policy.These examples are shown in Figure B.Source: §3.3 · p. 7

4 Experiments

Experiments compare open- and closed-source LMMs on JMMMU-Pro and JMMMU, using Direct and Chain-of-Thought prompts for open-source models. Open-source models score poorly on JMMMU-Pro, while closed-source models perform substantially better.

4.1 Setup

  • Evaluation covered English-centric, multilingual, and Japanese open-source LMMs; open-source results use the better of Direct and CoT prompting, while closed-source models use Direct prompting.The closed-source models evaluated were GPT-5.2 and Gemini3Pro.Source: §4.1 · pp. 7–8

4.2 Main Results

  • 45.83 was the best open-source JMMMU-Pro score, achieved by Qwen3-VL-8B; nine models scored below 32%.The results indicate substantial room for improvement among open-source LMMs.Source: §4.2 · p. 8
  • Most open-source models, except Qwen2.5-VL-7B, lost accuracy versus JMMMU; models with JMMMU CS/CA gaps were similarly low on both JMMMU-Pro subsets.The authors interpret this pattern as evidence of a fundamental vision-side understanding weakness rather than a question-type weakness.Source: §4.2 · p. 8
  • Closed-source LMMs scored substantially higher than open-source models, exposing a gap in integrated Japanese visual-textual understanding.The authors describe reducing this gap as an important community goal.Source: §4.2 · p. 8

5 Analysis

The analysis finds that CoT benefits depend on both model and benchmark, while OCR performance is positively associated with JMMMU-Pro accuracy but does not fully explain it. Representative failures also involve reasoning and perception, pointing to integrated visual-textual demands.

5.1 Impact of CoT Prompting

  • On JMMMU-Pro, 7 of 12 LMMs improve with CoT, compared with 3 models on JMMMU; prompt preferences can differ by model and setting.The authors conclude that prompting strategies should be tailored to each model and task.Source: §5.1 · p. 8, Figure 5 · p. 9

5.2 Correlation with OCR Performance

  • OCR accuracy and JMMMU-Pro accuracy have a positive correlation of 0.593, but comparable OCR performance can accompany very different benchmark accuracy.The authors say solving JMMMU-Pro also requires integrated interpretation and reasoning across language and vision.Source: §5.2 · p. 9, Figure 6 · p. 9

5.3 Failure Examples

  • Representative Qwen3-VL-8B failures on JMMMU-Pro include both reasoning and perceptual errors, suggesting demands beyond simple OCR.The examples are presented in Figures C and D.Source: §5.3 · p. 9

6 Conclusion

The conclusion presents JMMMU-Pro as an image-based Japanese multimodal benchmark and Vibe Benchmark Construction as a scalable way to create such datasets. The authors report that open-source LMMs struggle on the benchmark and see it as useful for evaluation and future development.

6 Conclusion

  • JMMMU-Pro combines Japanese questions and images in an image-based benchmark, while Vibe Benchmark Construction is proposed as a scalable dataset-creation methodology.The conclusion frames the benchmark as a tool for more rigorous evaluation of Japanese multimodal capabilities and the construction method as a framework for future image-based VQA benchmarks.Source: §6 · pp. 9–10
  • The authors report that all open-source LMMs face significant difficulty on JMMMU-Pro, which they present as a benchmark for guiding future open-source progress.Source: §6 · pp. 9–10

A.1 Image Generation Failures in Nano Banana Pro

Nano Banana Pro sometimes produces image-generation failures, so the authors manually inspect and filter outputs. Some image characteristics are difficult to generate automatically, leaving a role for manual construction.

A.1 Image Generation Failures in Nano Banana Pro

  • Nano Banana Pro can produce failures attributed to generative-output diversity, making manual inspection and filtering important for correctness.The figure examples include incorrect image tags and incorrect text.Source: §A.1 · p. 14

A.2 Manual Construction Examples

  • Images with characteristics difficult for Nano Banana Pro to generate automatically were manually constructed, so the Vibe pipeline did not produce every image.Source: §A.2 · p. 14

B Verifying the original JMMMU Benchmark

The paper verifies and corrects the original JMMMU benchmark and adjusts its evaluation procedure. Its appendices also document prompts for image generation and LMM inference.

B Verifying the original JMMMU Benchmark

  • The authors corrected open-ended questions and sample errors, revised answer parsing, and named the resulting benchmark JMMMU-verified-2025-12; all reported JMMMU scores use this version.They manually checked generated false choices and changed failed parsing from random selection to marking the response incorrect.Source: B Verifying the original JMMMU Benchmark · pp. 14–15

C.1 Image Generation Prompt

  • Image-generation prompts ask Nano Banana Pro to arrange question content on a surface or screen and allow specifications for state, font, background, color, and margin.Source: §C.1 · pp. 15, 18

C.2 LMM Inference Prompt.

  • Inference prompts distinguish OCR extraction of question text and choices from solving the image-based question, with the OCR instruction excluding text from associated images.The solving prompt asks models to provide an answer choice; another prompt asks for step-by-step reasoning.Source: §C.2 · pp. 18–19

D Full Results

The full results are reported in separate tables for direct prompting and Chain-of-Thought prompting.

D Full Results

  • Tables A and B provide detailed results for the Direct Prompt and CoT Prompt, respectively.The supplied passage does not state any table values or model comparisons.Source: D Full Results · p. 19

E Failure Cases

The failure examples show Qwen3-VL-8B making perceptual and reasoning errors on JMMMU-Pro, beyond mere OCR capability.

E Failure Cases

  • Qwen3-VL-8B exhibits both reasoning and perceptual errors specifically on JMMMU-Pro.The examples are presented as representative failure cases.Source: E Failure Cases · p. 19
  • The examples indicate that JMMMU-Pro requires deeper integrated visual-textual understanding beyond mere OCR capability.Table A is identified as reporting direct-prompt results, but no values are supplied here.Source: E Failure Cases · pp. 19–20

Japanese Open LMMs

The supplied materials identify a perceptual-error example and a reasoning-error example, alongside a table of CoT-prompt results.

Japanese Open LMMs

Loading 2512.14620v1…