Source-linked AI summary
BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation
Yan Li, Zezi Zeng, Ziwei Zhou, Xin Gao, Muzhao Tian, Yifan Yang, Mingxi Cheng, Qi Dai, Yuqing Yang, Lili Qiu, Zhendong Wang, Zhengyuan Yang, Xue Yang, Lijuan Wang, Ji Li, Chong Luo
TL;DR
Existing benchmarks do not adequately evaluate the structured, multi-constraint demands of commercial visual design. BizGenEval introduces a cross-domain, checklist-based benchmark for this purpose, and its large-scale evaluation reveals substantial capability gaps between current image generation systems and professional requirements.
Problem
Existing benchmarks mainly emphasize natural images or simplified capabilities, leaving commercial visual generation without comprehensive standardized evaluation.
Method
BizGenEval evaluates 5 commercial document domains across 4 capability dimensions using 400 prompts and checklist-based MLLM assessment.
Results
The benchmark reveals substantial performance differences across domains and capability dimensions, with current systems showing gaps relative to real-world commercial design requirements.
Takeaways & Limitations
BizGenEval provides a standardized framework for analyzing strengths and limitations of image generation systems in practical commercial visual content creation.
Abstract
from arXiv · showhide
Recent advances in image generation models have expanded their applications beyond aesthetic imagery toward practical visual content creation. However, existing benchmarks mainly focus on natural image synthesis and fail to systematically evaluate models under the structured and multi-constraint requirements of real-world commercial design tasks. In this work, we introduce BizGenEval, a systematic benchmark for commercial visual content generation. The benchmark spans five representative document types: slides, charts, webpages, posters, and scientific figures, and evaluates four key capability dimensions: text rendering, layout control, attribute binding, and knowledge-based reasoning, forming 20 diverse evaluation tasks. BizGenEval contains 400 carefully curated prompts and 8000 human-verified checklist questions to rigorously assess whether generated images satisfy complex visual and semantic constraints. We conduct large-scale benchmarking on 26 popular image generation systems, including state-of-the-art commercial APIs and leading open-source models. The results reveal substantial capability gaps between current generative models and the requirements of professional visual content creation. We hope BizGenEval serves as a standardized benchmark for real-world commercial visual content generation.
1 Introduction
Image generation models are increasingly used for practical commercial visual content, but existing benchmarks do not systematically test the dense, multi-constraint requirements of these tasks. BizGenEval addresses this gap with a benchmark spanning professional document types, capability dimensions, curated prompts, and checklist-based evaluation.
- Commercial image generation increasingly targets slides, webpages, scientific figures, posters, and charts beyond aesthetic imagery.
- Existing benchmarks largely focus on natural images, object compositionality, aesthetic quality, or simplified text rendering rather than holistic commercial visual generation.
- BizGenEval covers 5 document domains and 4 capabilities—Text Rendering, Layout Control, Attribute Binding, and Knowledge-based Reasoning—forming 20 evaluation tasks.
- The benchmark constructs realistic prompts from professional commercial references and extends evaluation to factual and conceptual knowledge across physics, chemistry, mathematics, history, and arts.
- An MLLM-based judging pipeline answers task-specific verification questions, with human evaluation showing strong agreement between MLLM and human judgments.
- Large-scale evaluation reveals substantial performance differences across domains and capability dimensions, exposing strengths and limitations of current systems.
- 400 curated prompts and 8,000 human-verified checklist questions support rigorous assessment of complex visual and semantic constraints.
2 Related Work
Prior benchmarks address individual commercial document types or fundamental image-generation capabilities, whereas BizGenEval evaluates commercial visual generation holistically across multiple domains and dimensions.
- Existing commercial-document benchmarks are often domain-specific, covering areas such as slides or scientific figures rather than a broad range of professional artifacts.
- BizGenEval provides a holistic perspective across slides, charts, webpages, posters, and scientific figures while also evaluating core capabilities such as text rendering.
3 BizGenEval
BizGenEval organizes commercial visual generation into a 5-by-4 task space, builds its data from curated professional references and hidden knowledge rationales, and evaluates outputs with human-designed checklists and MLLM judges. Its statistics and scoring design target fine-grained visual and semantic constraints while separating easy and hard requirements.
- Task Description: BizGenEval crosses 5 commercial content domains with 4 capability dimensions, producing 20 evaluation tasks.
- Task Description: The domains include webpages, slides, charts, scientific figures, and posters, while capabilities cover layout, attribute, text, and knowledge control.
- Benchmark Construction: The benchmark collects 1,819 professional design candidates, filters them through human review, and retains 20 representative references per domain–task combination.
- Benchmark Construction: Knowledge-based prompts hide key facts as evaluation rationales to prevent answer leakage and encourage models to rely on internal knowledge.
- Benchmark Construction: 400 generation samples are paired with 8,000 binary checklist questions, with each task containing 10 easy and 10 hard items.
- Benchmark Statistics: Prompt statistics and keyword distributions align with targeted visual or semantic elements, supporting the checklist design’s intended coverage of fine-grained constraints.
- Evaluation Criteria: An MLLM evaluates generated images by answering each checklist question, and scores are reported separately for easy and hard tracks.
- Evaluation Criteria: The track score is max(0, 1−αN_errors) with α = 0.2, so five errors among ten questions produce a zero score.
4 Evaluation Results
BizGenEval evaluates 26 image generation models across five commercial content domains and reveals substantial performance differences by domain and capability. Closed-source models generally outperform open-source systems, while precise charts, scientific figures, layout control, and attribute binding remain challenging.
- Experimental Setup: 26 models, including 10 closed-source and 16 open-source systems, are evaluated using official default inference settings.The evaluated systems include commercial APIs and leading open-source models.
- Content Domains: Closed-source models achieve their strongest results on Slides, Webpages, and Posters, whereas Charts and Scientific Figures produce substantially lower scores.Hard-set averages for leading closed-source models are around 68–77%, while GPT-Image-1.5 falls to 28.2–27.8% on difficult chart and scientific-figure cases.
- Content Domains: Specific-value chart generation separates models: Nano-Banana-Pro correctly plots and labels values, GPT-Image-1.5 homogenizes them as “12,” and Qwen-Image-2512 omits the markers.The task requires markers 14, 13, 12, 11, and 12 at a CLIP Score of 0.24.
- Capability Dimensions: Text Rendering and Knowledge-based Reasoning are comparatively strong for closed-source models, while Layout Control and Attribute Binding remain difficult, especially on hard subsets.Nano-Banana-Pro scores 86.4/95.0 on Text Rendering and 82.6/96.2 on Knowledge-based Reasoning, versus 72.2 for Layout and 65.6 for Attribute on cited hard subsets.
- Capability Dimensions: Qualitative examples show that even top-tier models fail highly specific spatial constraints, such as placing panel labels strictly inside panel corners.Nano-Banana-Pro produces more coherent structures but still fails the required boundary logic in the cited layout task.
- Evaluation Reliability: Automated evaluation agrees strongly with human judgments, reaching po = 90.88% agreement and Cohen’s κ = 0.7692 after removing nine quality-check failures.The human evaluation used participants with prior experience in visual design or data interpretation.
5 Conclusion
BizGenEval is introduced as a systematic benchmark for commercial visual content creation, spanning five document types and four capability dimensions. Large-scale evaluation reveals substantial gaps between current models and real-world commercial design requirements.
- BizGenEval evaluates commercial visual content generation across five document types and four capability dimensions.The framework combines diverse tasks with checklist-based verification.
- Large-scale benchmarking of commercial APIs and open-source models reveals substantial gaps in structured layout, precise attribute control, and complex multi-constraint generation.
- BizGenEval is intended to serve as a standardized benchmark for practical visual content generation and design-aligned model development.
Supplemental Materials for BizGenEval
The supplemental materials provide examples, rankings, evaluator analyses, and qualitative studies covering BizGenEval’s domains and capability dimensions. They also include performance ladders that organize model capabilities and the difficulty of commercial design tasks.
- Performance rankings: Supplemental rankings report model performance across five content domains and four capability dimensions.The materials organize rankings by application domain and fundamental capability.
- Supplemental examples: The supplemental materials include visual examples of prompts and human-verified checklist questions across four capability dimensions.The examples cover Knowledge-based Reasoning, Attribute Binding, Layout Control, and Text Rendering.
- Performance rankings: Figure 10 presents an overall performance ladder, while Figures 11–15 and 16–19 show rankings by domain and capability dimension.
- Interpretation: The rankings establish a hierarchy of current state-of-the-art models and highlight the relative difficulty of realistic commercial design tasks.
B.1 Comparison across content domains
The benchmark compares models across Webpage, Slides, Chart, Poster, and Scientific Figure scenarios by their ability to handle domain-specific constraints. Rankings distinguish performance on hard and easy subsets.
- Models are ranked on domain-specific constraints in Webpage, Slides, Chart, Poster, and Scientific Figure scenarios.These scenarios correspond to Figures 11–15, respectively.
- Darker bars represent hard-subset performance, while lighter bars represent easy-subset scores.
B.2 Comparison across capability dimensions
BizGenEval analyzes model strengths across Layout Control, Attribute Binding, Text Rendering, and Knowledge-based Reasoning. Its knowledge-reasoning task uses a multi-panel Bayes’ theorem figure with explicit probability and visualization requirements.
- Capability dimensions: Performance is evaluated across Layout Control, Attribute Binding, Text Rendering, and Knowledge-based Reasoning.
- Knowledge-based Reasoning: The Knowledge-based Reasoning task requires a publication-style six-panel figure explaining Bayes’ theorem and the base rate fallacy.
- Knowledge-based Reasoning: The task specifies probability trees, formal equations, marginal-probability calculations, numeric substitution, a final percentage, and a visual demonstration of false positives.
C.1 Human-Alignment and Case Studies
The selected MLLM evaluator aligns closely with expert human judgments, supporting its use for checklist-based assessment. Qualitative cases show that it can verify complex spatial relationships and exact element counts.
- Gemini-3-Flash achieves the highest agreement and Cohen’s Kappa among the evaluated MLLM judges.The comparison is reported for the checklist-based protocol against expert human annotations.
- The selected evaluator accurately verifies complex spatial constraints, including a two-line caption block placed below specified scene levels.
- The evaluator precisely counts chart elements, verifying 13 vertical bars against 13 categories on the x-axis.
C.2 Stability Analysis
The checklist-based evaluation protocol is reported as stable across repeated trials, with minimal variance when Gemini-3-Flash evaluates selected models.
- Minimal variance across three independent evaluation passes confirms the stability of the checklist-based scoring system.Gemini-3-Flash is used as the judge in this stability test.
- Table 4 compares overall agreement and Cohen’s Kappa across different MLLM judges in the human-alignment analysis.
- Table 5 reports evaluator stability as variance across three independent evaluation trials for selected models.