PaperlayerView exact revision on arXiv

Source-linked AI summary

GenExam: A Multidisciplinary Text-to-Image Exam

Zhaokai Wang, Penghao Yin, Xiangyu Zhao, Changyao Tian, Yu Qiao, Wenhai Wang, Jifeng Dai, Gen Luo

arXiv:2509.14232v5cs.CV

TL;DR

Existing benchmarks seldom evaluate rigorous multidisciplinary drawing exams, although solving exams calls for integrated understanding, reasoning, and generation. GenExam pairs exam-style prompts with reference images and fine-grained criteria; experiments show a marked performance gap between leading closed-source models and open-source models.

  • Problem

    Multidisciplinary benchmarks mainly test understanding, while image-generation benchmarks emphasize world knowledge or concept illustration rather than rigorous drawing exams.

    Source: §1 · p. 1

  • Method

    GenExam provides multidisciplinary exam-style prompts with ground-truth images and tailored scoring points to assess semantic correctness and visual plausibility.

    Source: §1 · p. 2

  • Results

    Nano Banana Pro and GPT-Image-1.5 score 70.2% and 42.5% strictly, while open-source Qwen-Image-2512 and FLUX.2 dev score below 5%.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Experiments on 17 models demonstrate GenExam’s difficulty and a gap between closed-source and open-source models in multidisciplinary text-to-image exams.

    Source: §5 · p. 9

Abstract

from arXiv · show

Exams are a fundamental test of expert-level intelligence and require integrated understanding, reasoning, and generation. Existing exam-style benchmarks mainly focus on understanding and reasoning tasks, and current generation benchmarks emphasize the illustration of world knowledge and visual concepts, neglecting the evaluation of rigorous drawing exams. We introduce GenExam, the first benchmark for multidisciplinary text-to-image exams, featuring 1,000 samples across 10 subjects with exam-style prompts organized under a four-level taxonomy. Each problem is equipped with ground-truth images and fine-grained scoring points to enable a precise evaluation of semantic correctness and visual plausibility. Experiments on 17 text-to-image and unified models demonstrate the great challenge of GenExam and the huge gap where open-source models consistently lag behind the leading closed-source ones. By framing image generation as an exam, GenExam offers a rigorous assessment of models' ability to integrate understanding, reasoning, and generation, providing insights for on the path to intelligent generative models. Our benchmark and evaluation code are released at https://github.com/OpenGVLab/GenExam.

1. Introduction

Existing benchmarks rarely evaluate multidisciplinary image generation as a rigorous drawing exam, despite exams requiring integrated understanding, reasoning, and generation. GenExam introduces such a benchmark with curated prompts, reference images, and fine-grained criteria; experiments reveal substantial challenges for current models.

1. Introduction

  • GenExam fills a gap left by multidisciplinary benchmarks focused on understanding and generation benchmarks centered on world knowledge or concept illustration rather than rigorous drawing exams.Exam-style prompts add complex constraints, reference answers, scoring criteria, and broad knowledge organized hierarchically.Source: §1 · p. 1
  • GenExam introduces 1,000 multidisciplinary exam-style prompts across 10 subjects, with a four-level taxonomy, ground-truth images, and carefully reviewed requirements.Prompts are developed through GPT-5 and human collaboration in the style of real exams.Source: §1 · p. 2
  • Its sample-specific scoring points assess semantic correctness, while spelling, readability, and logical consistency capture visual plausibility.The evaluation uses ground-truth images as references and reports strict and relaxed scores.Source: §1 · p. 2
  • Nano Banana Pro and GPT-Image-1.5 achieve strict scores of 70.2% and 42.5%, respectively, while most models score below 15% and some latest models below 3%.The reported results highlight a large gap between closed-source and open-source models.Source: §1 · p. 2

2. Related Work

Prior image-generation approaches and benchmarks cover visual quality, prompt alignment, and reasoning-informed generation, but leave rigorous multidisciplinary drawing exams largely untested. GenExam targets that gap with precise prompts and fine-grained scoring.

2.1. Generative Models

  • Diffusion and autoregressive image generators emphasize aesthetic quality and semantic consistency, while unified MLLMs combine multimodal understanding with image generation.Despite strong natural-image results from leading models, their multidisciplinary exam performance remains limited.Source: §2.1 · pp. 2–3

2.2. Image Generation Benchmarks

  • Image-generation evaluation has progressed from image-similarity metrics to prompt-image alignment and reasoning-informed MLLM judging, often focused on world knowledge or commonsense.These benchmarks commonly use natural or synthetic images.Source: §2.2 · p. 3
  • Existing multidisciplinary image-generation benchmarks mainly test concept illustration with loose criteria, whereas GenExam uses complex exam prompts and fine-grained scoring for rigorous evaluation.GenExam contains examples pairing exam-style prompts with reference images.Source: §2.2 · p. 3, Figure 2 · p. 3

3. GenExam

GenExam combines a multidisciplinary exam-style dataset with sample-specific scoring criteria and a four-level taxonomy. Its evaluation judges semantic correctness and visual plausibility, then reports strict and relaxed scores.

3.1. Overview

  • GenExam contains 1,000 samples across 10 subjects, each with a ground-truth image, a four-level taxonomy, and several scoring points.The taxonomy includes subject-specific paths such as Mathematics/Plane Geometry/Circles/Tangent and is based on ISCED-F fields.Source: §3.1 · p. 4
  • Prompts average 74.8 words and 6.9 scoring points, while the image-type distribution illustrates benchmark diversity.A subset is also provided for efficient evaluation.Source: §3.1 · p. 4, Table 1 · p. 5, Figure 3 · p. 5
  • GPT-5 filters collected images and drafts prompts and scoring points, which undergo rigorous human refinement.Filtering rates text richness, image domain, complexity, and subject-knowledge density.Source: §3.1 · p. 4

3.2. Evaluation Protocol

  • For semantic correctness, an MLLM judge answers each sample’s scoring-point questions yes or no, with the ground-truth image available as a reference; scores range from 0 to 1.The scoring-point weights sum to 1, and each point is designed to test a specific prompt requirement.Source: §3.2 · p. 4, §3.1 · p. 4
  • Visual plausibility scores spelling, logical consistency, and readability from 0 to 2, with higher scores indicating fewer flaws.The dimensions assess rendered text, internal consistency of marks and labels, and whether components remain readable and identifiable.Source: §3.2 · pp. 4–5
  • The strict score counts only images with perfect semantic correctness and top scores across all three plausibility dimensions; the relaxed score is their weighted average.The relaxed-score weights are 0.7 for semantic correctness and 0.1 each for spelling, logical consistency, and readability; GPT-5 is the main judge, with other MLLMs also tested for robustness.Source: §3.2 · p. 5, Figure 4 · p. 5

4. Experiments

Experiments compare 17 representative text-to-image and unified models, assess judge alignment with human ratings, and analyze model failures. Results show a strict-score gap between leading closed-source models and open-source systems, alongside differences in how models handle semantic correctness and visual quality.

4.2. Main Results

  • 70.2% and 42.5% strict scores for Nano Banana Pro and GPT-Image-1.5 contrast with less than 5% for Qwen-Image-2512 and FLUX.2 dev.The table reports strict and relaxed scores on GenExam.Source: Table 2 · p. 6, §4.2 · p. 6

4.1. Baselines

  • Seventeen representative models span proprietary text-to-image systems, open-source text-to-image models, and open-source unified multimodal models, each run with its default configuration.The experiment also evaluates judge alignment and analyzes failure cases.Source: §4 · p. 5, §4.1 · p. 6

4.2. Main Results

  • 42.3 and 35.3 relaxed scores put FLUX.2 dev and Qwen-Image-2512 atop open-source methods, while BAGEL and Show-o2 lag dedicated text-to-image models.The open-source/closed-source gap persists under relaxed scoring.Source: §4.2 · p. 6

4.3. Human Alignment

  • Low MAE and significant correlations (p < 0.05) support judge alignment; GPT-5 relaxed scores slightly exceed semantic-only, unlike CLIP and VQA, while Gemini-3-Flash also aligns strongly.The evaluation compares automatic judgments with expert scores across four dimensions.Source: §4.3 · p. 7

4.4. Error Analysis

  • Closed-source leaders outperform across all four dimensions, while open-source models especially lag in spelling and logic; some failures occur despite identifying correct disciplinary content.The FLUX.2 dev history example identifies regions correctly but places graphics incorrectly.Source: §4.4 · pp. 7–8
  • Fine-grained scoring captures nuanced visual flaws that broad photorealism or aesthetic measures would typically overlook.The reported failures include legibility errors, logical inconsistencies, and spelling inaccuracies.Source: §4.4 · p. 8

5. Conclusion

GenExam is presented as a benchmark for testing whether models integrate understanding, reasoning, and generation on multidisciplinary graph-drawing problems. Experiments on 17 models show benchmark difficulty and a gap between closed-source and open-source systems.

5. Conclusion

  • GenExam is presented as the first benchmark for multidisciplinary text-to-image exams, testing integrated understanding, reasoning, and generation on graph-drawing problems.It includes diverse, challenging prompts, ground-truth images, and fine-grained scoring points.Source: §5 · p. 9
  • Experiments on 17 existing models demonstrate the benchmark's difficulty and a performance gap between closed-source and open-source models.Source: §5 · p. 9
  • The authors hope GenExam will guide generative models toward expert-level intelligence in a multidisciplinary direction.Source: §5 · p. 9

A. More Statistics

The appendix describes GenExam's taxonomy and sample distributions. It also reports diversity in image sources and subject-specific prompt vocabulary.

A. More Statistics

B. Data Curation Pipeline

The data pipeline uses a four-level taxonomy to collect and filter candidate images, then generates exam-style prompts and scoring points. Human review refines these materials before the final dataset is selected.

B. Data Curation Pipeline

  • A manually checked four-level taxonomy guides keyword generation and collection of web images and images from subject-related datasets.The sources provide 40K images with original text descriptions for initial processing.Source: B. Data Curation Pipeline · p. 15
  • Filtering removes duplicates, low-resolution images, watermarks, and non-English text, while GPT-5 rates text richness, image domain, complexity, and subject-knowledge density.The filtering criteria exclude text-dominated images, undesirable domains, excessive complexity, and low subject-knowledge density.Source: B. Data Curation Pipeline · p. 15
  • Automatic filtering yields 6.5K images for annotation.Source: B. Data Curation Pipeline · p. 15
  • GPT-5 drafts prompts and scoring questions from source descriptions, with prompt design targeting exam-like knowledge and reasoning and scores weighted by question difficulty.Questions should be neither overly general nor too complex to split into subquestions.Source: B. Data Curation Pipeline · p. 15
  • Three PhD annotators inspect images, prompts, and scoring points; manual edits and cross-checks produce the final 1K-image dataset.Reviewers remove unsuitable images and revise prompts or scoring points to match the ground truth.Source: B. Data Curation Pipeline · pp. 15–16

C.1. Ablation on Evaluator Models

The evaluator comparison favors GPT-5 with low reasoning effort, while open-source evaluators lag and ground-truth image references improve evaluation precision.

C.1. Ablation on Evaluator Models

  • The evaluator study compares models on scoring-point accuracy, visual-plausibility MAE, and Kendall’s τ, Spearman’s ρ, and Pearson’s r.The comparison uses the same manually evaluated samples; Table 5 is labeled as human alignment on images generated by more models.Source: §C.1 · p. 16, Table 5 · p. 17
  • GPT-5 with low reasoning effort achieves the best evaluator performance at acceptable cost and time; medium or high effort may lower performance and raise costs.Qwen3-VL performs relatively strongly among open-source evaluators but still trails GPT-5.Source: §C.1 · p. 16
  • Open-source evaluators lag behind GPT-5, and some show near-random scoring-point accuracy or correlations without statistical significance.The authors attribute these weaknesses to the difficulty of multidisciplinary images for current multimodal understanding models.Source: §C.1 · p. 16
  • Ground-truth images as references improve evaluation precision; text-only references perform slightly worse, while combining text and images adds no significant improvement.The authors cite spatial and detail fidelity and lower ambiguity as advantages of image references.Source: §C.1 · p. 17

C.2. Detailed Comparison of Existing Methods

This section provides detailed comparisons of existing methods alongside taxonomy and subject-level evaluation details. It covers human-alignment scores across models, performance variation by subject-knowledge difficulty and image type, and topic labels and sample counts across several disciplines.

C.2. Detailed Comparison of Existing Methods

  • Relaxed scores correlate strongly with human ratings across five additional models, and standard deviations indicate high inter-rater reliability.The comparison extends the human-alignment experiment to 500 additional samples.Source: §C.2 · p. 17
  • GenExam-Mini model performance is consistent with the full dataset, supporting its use as a valid subset for efficient, affordable validation.The comparison appears in Table 6(a).Source: §C.2 · p. 17, Table 6 · p. 18
  • Models perform worse on samples with higher subject-knowledge difficulty, highlighting the challenge of integrating multidisciplinary knowledge into generation.These results are categorized by difficulty level in Table 6(b).Source: §C.2 · p. 17, Table 6 · p. 18
  • Models score lower on some image types, including chemical structures, geometric shapes, sheet music, and trees and graphs.The authors identify these categories as potential directions for future improvement.Source: §C.2 · p. 17, Table 7 · p. 19
  • Table 9 reports semantic correctness, spelling, logical consistency, and readability scores for each subject.The passage provides the dimensions and subject-level scope, but no score values.Source: §C.2 · p. 17, Table 9 · p. 21

D. Full Taxonomy List

  • The appendix lists the four-level taxonomy with sample counts and indicates its mapping to UNESCO ISCED-F codes.Blue text marks the ISCED-F mapping.Source: D. Full Taxonomy List · p. 22
  • Math topics include definite-integral area, exponential and logarithmic functions, derivative geometry, inequality regions, linear programming, parametric and polar curves, and trigonometric functions.The listed counts range from 1 to 12 across these entries.Source: D. Full Taxonomy List · p. 25
  • Geometry entries cover inscribed and circumscribed circles, complex geometry, rectangles and polygons, perpendicular bisectors, pyramids, and straight cuts.The listed counts include 23 complex-geometry problems and 25 rectangle-and-polygon problems.Source: D. Full Taxonomy List · p. 25
  • Music topics include perfect fifths and major thirds, chord structures and interval diagrams, triads and seventh chords, and key and time signatures.The inventory lists 65 music samples overall.Source: D. Full Taxonomy List · p. 25
  • Circuits and electronics entries include DC and AC circuits, Kirchhoff-law applications, component symbols and combinations, signals, and electric and magnetic fields.The overall circuits-and-electronics category contains 36 samples.Source: D. Full Taxonomy List · p. 26
  • Physics entries include electromagnetic induction, energy loss and hysteresis loops, interference and diffraction, reflection and refraction, and intermolecular forces.The taxonomy also identifies physics and electronics-and-automation category labels.Source: D. Full Taxonomy List · p. 26
  • Additional physics topics are molecular speed distributions and pressure-volume-temperature relationships.The inventory lists 1 molecular-speed-distribution sample and 2 pressure-volume-temperature samples.Source: D. Full Taxonomy List · p. 26

E. Prompts

The appendix specifies how images are filtered and prompts and scoring points are constructed, then gives instructions for evaluating generated images against exam criteria and global quality dimensions.

E.1. Automatic Filtering

  • Automatic filtering rates image text richness, domain, complexity, and subject knowledge, with defined scoring scales for each dimension.Natural or pathological images receive a domain score of 0; other images receive 1.Source: §E.1 · p. 27

E.2. Annotating

  • Prompt generation targets academic-style multidisciplinary drawing questions that require subject knowledge and reasoning rather than fully describing the image.The prompt is intended to correspond to the image and taxonomy while leaving knowledge-dependent content to be inferred.Source: §E.2 · p. 28
  • Prompts should be concise, in English, and under 200 words; source conversations and image text may help but can contain irrelevant information.Source: §E.2 · p. 28
  • Scoring points are yes-or-no questions grounded in the prompt, with specific, decomposed criteria rather than broad judgments and no more than 20 questions.A correct image is expected to satisfy all points.Source: §E.2 · p. 28
  • Scoring-point weights sum to 1, with larger weights assigned to complex, knowledge-dependent, or detailed criteria than to basic structure checks.The benzene example divides weight equally across five checks of atom counts, bond counts, and atom colors.Source: §E.2 · p. 28

E.3. Evaluation

  • Evaluation considers the generated image only, using the ground-truth image as a reference while reasoning through each scoring point before assigning a binary answer.The evaluator is also instructed to describe the generated image’s components first.Source: §E.3 · p. 29
  • Global evaluation assigns 0–2 scores for spelling, readability, and logical consistency, including labels, marks, equations, and data relationships.Readability checks whether components and labels are clear and properly placed; logical consistency checks whether marks and information agree.Source: §E.3 · p. 29
  • The rubrics reserve score 0 for critical problems and score 2 for perfect or nearly perfect performance, allowing tiny errors.This grading pattern is specified for readability, logical consistency, and spelling.Source: §E.3 · p. 29

F. More Visualization

The section provides additional generated-image visualizations alongside examples of multidisciplinary prompts and their fine-grained scoring criteria.

F. More Visualization

  • Figures 11–13 provide additional visualizations of generated images.The supplied text identifies the figures but does not describe comparative outcomes.Source: F. More Visualization · p. 30, Figure 11 · p. 30, Figure 12 · p. 31
  • A Nunavut tundra food-web prompt specifies organisms and directed food-to-consumer links, while its rubric checks the listed links and excludes additional ones.The example is categorized under Biology/Ecology/Ecosystem/Food Chain and Food Web.Source: F. More Visualization · p. 31
  • A karst-landform scoring point requires a lateral bedrock cross-section rather than a plan view or non-karst landscape.The example is categorized under Geography/Earth Science/Landforms and Geology.Source: F. More Visualization · p. 31
  • A galvanic-cell task specifies zinc and chromium half-cells, a KNO3 salt bridge, and a voltmeter, with separate scoring checks for their components and labels.The rubric assigns individual scores to the half-cells, salt bridge, external circuit, and solution details.Source: F. More Visualization · p. 32
  • A mechanics prompt asks for five rods illustrating compression, tension, shear, torsion, and bending, with rubric points checking each loading pattern and arrow direction.The taxonomy places this example under Engineering/Mechanics of Materials/Stress Strain Relationship.Source: F. More Visualization · p. 32

Related papers

Loading 2509.14232v5…