Source-linked AI summary
IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation
Yinghao Tang, Xueding Liu, Boyuan Zhang, Tingfeng Lan, Yupeng Xie, Jiale Lao, Yiyao Wang, Haoxuan Li, Tingting Gao, Bo Pan, Luoxuan Weng, Xiuqi Huang, Minfeng Zhu, Yingchaojie Feng, Yuyu Luo, Wei Chen
TL;DR
Reliable text-to-infographic generation remains difficult because plausible-looking outputs can contain distorted data encoding or incorrect text. IGENBENCH addresses this gap with a curated benchmark and atomic MLLM-verified questions, finding strong question-level performance but limited end-to-end correctness. The benchmark also identifies data-related dimensions as universal bottlenecks while focusing reliability evaluation separately from aesthetics, accessibility, and communicative effectiveness.
Problem
Existing evaluations do not provide a dedicated, interpretable benchmark for reliable infographic generation, despite T2I outputs containing overlooked data and textual errors.
Method
IGENBENCH uses 600 curated cases spanning 30 infographic types and evaluates generated outputs through 10 question types comprising atomic yes/no questions verified by MLLMs.
Results
The evaluation of 10 state-of-the-art T2I models reveals a three-tier hierarchy, with Nanobanana-Pro reaching 0.90 Q-ACC but only 0.49 I-ACC, while data-related dimensions remain universal bottlenecks.
Takeaways & Limitations
Current T2I models are not yet reliable for autonomous infographic generation, and data completeness and encoding are especially challenging.
Takeaways & Limitations
IGENBENCH evaluates semantic consistency and data encoding but does not assess communicative effectiveness, accessibility, or visual aesthetics.
Abstract
from arXiv · showhide
Infographics are composite visual artifacts that combine data visualizations with textual and illustrative elements to communicate information. While recent text-to-image (T2I) models can generate aesthetically appealing images, their reliability in generating infographics remains unclear. Generated infographics may appear correct at first glance but contain easily overlooked issues, such as distorted data encoding or incorrect textual content. We present IGENBENCH, the first benchmark for evaluating the reliability of text-to-infographic generation, comprising 600 curated test cases spanning 30 infographic types. We design an automated evaluation framework that decomposes reliability verification into atomic yes/no questions based on a taxonomy of 10 question types. We employ multimodal large language models (MLLMs) to verify each question, yielding question-level accuracy (Q-ACC) and infographic-level accuracy (I-ACC). We comprehensively evaluate 10 state-of-the-art T2I models on IGENBENCH. Our systematic analysis reveals key insights for future model development: (i) a three-tier performance hierarchy with the top model achieving Q-ACC of 0.90 but I-ACC of only 0.49; (ii) data-related dimensions emerging as universal bottlenecks (e.g., Data Completeness: 0.21); and (iii) the challenge of achieving end-to-end correctness across all models. We release IGENBENCH at https://igen-bench.vercel.app/.
1 Introduction
IGENBENCH addresses the unresolved reliability of T2I-generated infographics, which can look plausible while containing misleading data or textual errors. It introduces a benchmark and interpretable question-based evaluation, then shows substantial gaps between question-level and complete-infographic correctness.
- Motivation: IGENBENCH targets a missing evaluation setting: reliable text-to-infographic generation, where existing benchmarks and holistic chart scores do not identify specific errors.Prior evaluations focus on related tasks, prompt adherence, or holistic scoring rather than fine-grained infographic reliability.
- Benchmark: 600 curated test cases span 30 infographic types, supporting evaluation across diverse real-world generation scenarios.The benchmark is presented as the first comprehensive evaluation resource for infographic generation fidelity.
- Evaluation framework: Atomic yes/no questions across 10 question types enable fine-grained verification of visual elements, semantic consistency, and data encoding.The framework derives questions from prompt constraints and expert-informed dimensions such as data completeness and ordering.
- Findings: 10 state-of-the-art T2I models reveal a three-tier performance hierarchy and universal difficulty with data-related dimensions.The experiments examine current model capabilities and expose recurring reliability bottlenecks.
- Findings: 0.90 Q-ACC for Nanobanana-Pro contrasts with 0.49 I-ACC, showing that fewer than half of its infographics satisfy all specified constraints.The gap indicates that strong question-level performance does not ensure complete infographic correctness.
2 Related Work
Prior infographic-generation approaches include code-based pipelines and T2I systems, but the field lacks a dedicated benchmark for systematically evaluating T2I-generated infographics. Existing benchmarks mainly assess prompt following, aesthetics, chart legality, or holistic visualization quality.
- Infographics Generation: Code-based infographic generation executes visualization code but depends on manual templates and asset libraries, limiting scalability and generalizability.These approaches also struggle to freely render pictograms, thematic icons, and metaphorical imagery.
- Infographics Generation: T2I approaches have become a promising mainstream paradigm for infographic creation, including work on semantic context and layout-guided cross-attention.Recent progress in powerful T2I models has increased interest in this direction.
- Benchmarks: The field lacks a dedicated benchmark that systematically evaluates T2I-generated infographics.This gap motivates IGENBENCH.
- Benchmarks: Existing general T2I benchmarks primarily assess prompt-following ability or visual aesthetics rather than infographic reliability.The cited benchmarks include PartiPrompt, DrawBench, TIFA, T2I-CompBench, MJHQ-30K, and ArtiMuse.
- Benchmarks: Visualization benchmarks commonly use MLLMs for holistic scoring, while rule-based checks assess chart legality and related benchmarks study structured-image generation or editing.These efforts address adjacent tasks rather than providing a dedicated reliability benchmark for T2I-generated infographics.
3 Dataset Construction
IGENBENCH transforms diverse real-world infographic designs into self-contained generation prompts through collection, curation, and human-in-the-loop extraction. Its construction emphasizes authentic use cases and semantic and stylistic diversity.
- Pipeline: The dataset pipeline collects and curates real-world infographics, generates prompts, and supports later evaluation through a three-stage construction process.Figure 2 summarizes collection and curation, prompt generation, and question-based evaluation.
- Collection & Curation: 42,315 infographic charts were collected from Statista, Visual Capitalist, and the real-world portion of ChartGalaxy.Synthetic ChartGalaxy examples were excluded to focus on real-world usage scenarios.
- Collection & Curation: The taxonomy is refined by retaining clearly defined types, merging visually similar variants, and adding multi-panel layouts as a bonus category.This process produces six high-level categories and a fine-grained, interpretable taxonomy.
- Collection & Curation: MLLM-based assignment, intra-type deduplication, clustering, stratified sampling, and manual checks reduce semantic overlap and visual redundancy.Samples are filtered using similarity to cluster-center embeddings and quality review.
- Prompt Generation: A human-in-the-loop process extracts each infographic’s structural design description and underlying data table before prompt synthesis.The description captures layout, chart type, data encoding, text placement, and decorative elements without aesthetic details.
- Prompt Generation: The final self-contained T2I prompt fuses the verified design description with the data table so all required structural and numerical information is explicit.The prompt ends by embedding the data directly within the specification.
4 Evaluation Protocol
IGENBENCH evaluates infographic reliability by decomposing prompts into atomic verification questions and aggregating strict binary judgments at question and infographic levels. Its protocol combines prompt-derived and expert-informed checks across 600 cases and 5,259 verification questions.
- Question Set Construction: The protocol evaluates each generated infographic with atomic yes/no questions derived from prompt constraints and a taxonomy of 10 infographic elements.Questions target specific visual or textual requirements and are answerable by inspecting the generated infographic alone.
- Question Set Construction: Expert-informed questions additionally test Data Completeness, Data Ordering, and Data Encoding against visualization best practices.These requirements are instantiated as chart-specific checks and added to the prompt-derived question set.
- Verification: Each question receives a strict binary score, with ambiguity, partial satisfaction, or missing visual evidence counted as incorrect.A question scores 1 only when its requirement is clearly satisfied in the generated infographic.
- Reliability Metrics: Question-level accuracy measures satisfied verification questions across the evaluation set, whereas infographic-level accuracy requires every associated question for an infographic to be satisfied.The two aggregation levels distinguish partial correctness from complete infographic correctness.
- Benchmark Statistics: IGENBENCH contains 600 curated prompts and 5,259 verification questions, with most infographics evaluated using 7–11 questions.Its prompts are substantially longer than typical text-to-image benchmark prompts, reflecting infographic complexity.
5 Experiments
Experiments on 10 T2I models reveal a three-tier reliability hierarchy, persistent data-fidelity bottlenecks, and a large gap between component-level and end-to-end correctness. Automatic scores align strongly with human judgments, while benchmark rankings only moderately align with natural-image rankings.
- Overall Model Performance: Nanobanana-Pro leads with Q-ACC 0.90, while Seedream-4.5 and GPT-Image-1.5 reach 0.61 and 0.55, respectively, and remaining models fall below 0.5.The average Q-ACC across all models is 0.39.
- Data Fidelity: Data Completeness averages 0.21, Ordering 0.27, and Data Encoding 0.26, making data-related dimensions the most difficult across models.Nanobanana-Pro reaches 0.84 for Data Completeness and 0.86 for Data Encoding.
- Data Fidelity: Models perform better on layouts, chart types, titles, and decorative elements than on precise rendering and faithful encoding of underlying data values.Average scores for visual layouts, chart types, and titles are 0.49, 0.54, and 0.46, respectively.
- End-to-End Reliability: Nanobanana-Pro achieves Q-ACC 0.90 but I-ACC 0.49, showing that strong question-level performance does not ensure fully correct infographics.Seedream-4.5 and GPT-Image-1.5 fall to I-ACC 0.06 and 0.12, respectively.
- Evaluation Alignment: Automatic evaluation correlates strongly with human judgments at r = 0.90, whereas IGENBENCH and LMArena model rankings correlate at Spearman ρ = 0.78.The automatic-human correlation uses 100 bootstrap samples of 25 questions each.
- Case Study: A proportional-area case exposes simultaneous failures: Nanobanana-Pro generates 16 instead of 15 bubbles, while Seedream and Qwen-Image show ranking and text errors.These errors illustrate the difficulty of satisfying multiple fine-grained infographic constraints simultaneously.
6 Conclusion
IGENBENCH is introduced as the first benchmark for evaluating the reliability of text-to-infographic generation, using 600 curated cases and a question-driven framework. Evaluating 10 state-of-the-art T2I models reveals critical limitations in current infographic generation.
- IGENBENCH is the first benchmark for evaluating text-to-infographic generation reliability.It uses 600 curated test cases spanning 30 infographic types.
- The benchmark assesses 10 state-of-the-art T2I models through a question-driven evaluation framework.
7 Limitation
The study evaluates reliability through semantic consistency and accurate data encoding, rather than broader infographic qualities. It also limits model coverage because large-scale evaluation is costly.
- IGENBENCH focuses on semantic consistency and accurate encoding of underlying data values.It does not assess communicative effectiveness, accessibility, or visual aesthetics.
- Communicative effectiveness and accessibility are left for future work because they require different evaluation methodologies.The paper identifies user studies and perceptual modeling as examples of those methodologies.
- The evaluation includes only a selected set of representative state-of-the-art models because large-scale evaluation has high monetary cost.The authors plan to incorporate more models into the living benchmark.
8 Ethical Considerations
The study uses publicly available infographic images, manual review, expert input, human-in-the-loop prompt construction, and MLLM-based question answering. It addresses content safety and limits redistribution to image URLs.
- All infographic images are publicly available, manually reviewed, and released only through their URLs.The review checked for harmful, illegal, or sensitive content, and original images are not redistributed.
- Claude-4.5-Sonnet was used for English grammar polishing of the paper.
- MLLMs assisted with prompt synthesis in a human-in-the-loop process and automatically answered atomic verification questions.They were not used as autonomous decision makers during prompt construction.
- Expert discussions and targeted annotation contributed to benchmark design, dataset curation, and evaluation.Three coauthors developed the question taxonomy, while annotators filtered low-quality infographics.
C Benchmark Construction Detail
The benchmark constructs a diverse infographic dataset through stratified clustering, representative sampling, manual quality filtering, and expert-defined fidelity questions. Its taxonomy covers data-driven, design-oriented, and structural aspects of infographic correctness.
- Clustering & Sampling: The sampling algorithm applies k-means clustering to semantic embeddings separately for each chart type.It partitions samples into clusters and selects representative medoids, with optional additional sampling.
- Clustering & Sampling: The sampling procedure outputs selected sample indices after processing each chart type.
- Quality Filtering: Manual review filters samples with irrelevance, poor legibility, or insufficient informational content.The review also excludes incoherent chart combinations, low-resolution text or visuals, and pure charts lacking infographic narrative or design.
- Infographic Taxonomy: The benchmark organizes visualization types into six categories, including composition, categorical comparison, trends, deviation, correlation and flow, and multi-chart layouts.The listed categories include pie, bar, line, bubble, heatmap, alluvial, and multi-panel visualizations.
- Question Taxonomy: The question taxonomy evaluates both data-driven elements and design-oriented components of infographic fidelity.Data completeness and ordering additionally test whether elements and their structural relationships are preserved.
D.1 Performance Breakdown on Chart Type
Performance varies substantially by chart type: common visualizations are easier for models, while structurally complex charts remain difficult. Model rankings are generally stable across chart types despite occasional large shifts.
- Q-ACC varies markedly across chart types, with common visualizations outperforming structurally complex charts.Pie Chart, Bar Chart, and Line Graph achieve higher accuracy, whereas Alluvial Diagram, Radar Chart, Voronoi Treemap, and Bump Chart perform much worse.
- 0.53 average Q-ACC on Pie Chart versus 0.25 on Bump Chart illustrates the gap between simple and complex chart types.The difference reflects variation in structural complexity, spatial constraints, and encoding demands.
- 0.92 average Spearman rank correlation indicates largely stable model rankings across chart types.Some deviations remain: SeedDream-4.5 and GPT-Image-1.5 rank fourth on Pie Chart but seventh on Bump Chart.
D.2 Potential Data Leakage
The benchmark tests potential data leakage by comparing its original items with newer infographics created after evaluated models were released. The authors acknowledge that some leakage may remain and plan a live benchmark to address it.
- 100 post-December-2025 Visual Capitalist infographics were used to test potential data leakage.Because these samples postdate the evaluated models’ release dates, they could not have appeared in those models’ training data.
- Some degree of data leakage may still affect certain models, particularly those with more recent training cutoffs.The authors identify evolving IGENBENCH into a live benchmark as a future response.
D.3 Error Analysis of Automated Evaluation
The authors examine evaluator reliability, reference-based alternatives, prompt complexity, and question-type relationships. Results support decomposed QA evaluation while identifying data encoding disagreements, prompt data volume, and limited redundancy as important considerations.
- 12.12% disagreement makes Data Encoding the least reliable evaluated category for Gemini-2.5-Pro.The main error pattern is over-positive evaluation that misses subtle encoding violations, unlike near-perfect agreement for Title & Subtitle, Data Completeness, and Ordering.
- Evaluator rankings remain consistent across Gemini-2.5-Pro, GLM-4.5V, and GPT-5-Mini, with all pairwise Spearman ρ ≥0.95.Gemini-2.5-Pro did not favor Google generators, reducing concern that the main results reflect evaluator-specific provider bias.
- Reference-image metrics provide limited discrimination, while QA-based evaluation captures substantially more than visual resemblance.CLIP and LPIPS correlate moderately with Q-ACC, whereas SSIM and PSNR are near zero.
- ρ ≈−0.57 correlations show that longer prompts, especially those with more embedded data, reduce Q-ACC.GPT-Image-1.5 is most sensitive (ρ = −0.550), while Nanobanana-Pro is most robust (ρ = −0.198).
- 0.21 is the highest within-model correlation among question types, indicating that the ten dimensions capture largely distinct properties.The highest correlations involve Data Completeness with Data Encoding and Data Marks.