Source-linked AI summary
AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation
Zhaohe Liao, Kaixun Jiang, Zhihang Liu, Yujie Wei, Junqiu Yu, Quanhao Li, Hong-Tao Yu, Pandeng Li, Yuzheng Wang, Zhen Xing, Shiwei Zhang, Chen-Wei Xie, Yun Zheng, Xihui Liu
TL;DR
Academic illustration generation remains largely unexplored because producing paper-ready method figures requires long-context reasoning and dense visual generation, while holistic VLM evaluation is unreliable for complex logic. AIBench addresses this gap with a VQA-based benchmark built from logic graphs and multi-level questions, paired with model-based aesthetic assessment. Experiments reveal larger model differences than on general tasks, a logic–aesthetics trade-off, and gains from test-time scaling.
Problem
Academic illustration generation remains largely unexplored, while holistic VLM comparison is unreliable for long, complex texts and illustrations and does not clearly interpret component influence.
Method
AIBench constructs text-to-logic directed graphs and generates four-level QA pairs for visual-logical evaluation, alongside model-based aesthetic assessment.
Results
The performance gap between models is more profound on academic illustration generation than on general tasks, while logic and aesthetics exhibit a trade-off.
Takeaways & Limitations
Test-time scaling applied to reasoning or generation can effectively improve performance on complex academic illustration generation.
Takeaways & Limitations
The benchmark is curated from premier Artificial Intelligence conferences and lacks representation from disciplines with different diagrammatic conventions.
Abstract
from arXiv · showhide
Although image generation has boosted various applications via its rapid evolution, whether the state-of-the-art models are able to produce ready-to-use academic illustrations for papers is still largely unexplored. Directly comparing or evaluating the illustration with VLM is native but requires oracle multi-modal understanding ability, which is unreliable for long and complex texts and illustrations. To address this, we propose AIBench, the first benchmark using VQA for evaluating logic correctness of the academic illustrations and VLMs for assessing aesthetics. In detail, we designed four levels of questions proposed from a logic diagram summarized from the method part of the paper, which query whether the generated illustration aligns with the paper on different scales. Our VQA-based approach raises more accurate and detailed evaluations on visual-logical consistency while relying less on the ability of the judger VLM. With our high-quality AIBench, we conduct extensive experiments and conclude that the performance gap between models on this task is significantly larger than general ones, reflecting their various complex reasoning and high-density generation ability. Further, the logic and aesthetics are hard to optimize simultaneously as in handcrafted illustrations. Additional experiments further state that test-time scaling on both abilities significantly boosts the performance on this task.
1 Introduction
AIBench addresses the underexplored challenge of generating logically faithful academic illustrations and the limits of holistic VLM-based evaluation. It introduces a VQA-centered benchmark and finds substantial model differences, a logic–aesthetics trade-off, and benefits from scaling reasoning and generation.
- Motivation: Academic illustration generation remains largely unexplored because models must understand lengthy methodology text and render its core contribution as a framework figure.Existing academic illustrations still heavily rely on handcrafting.
- Motivation: Holistic VLM comparison can rely on questionable oracle understanding for complex papers and leaves component-level influence uninterpretable.These limitations motivate a more fine-grained evaluation design.
- Benchmark: AIBench evaluates logical consistency with multi-level VQA and aesthetics with model-based assessment, using questions that cover the underlying paper logic.The benchmark constructs text-to-logic directed graphs and annotates QA pairs across four hierarchical levels.
- Benchmark: AIBench contains 300 top-conference open-access papers and 5704 QA pairs checked by multiple human experts.The QA pairs target properties an ideal academic illustration should have.
- Findings: The performance gap between models is more profound on academic illustration generation than on general tasks, exposing demands for long-text reasoning and high-density generation.The experiments attribute the setting's difficulty to understanding complex text and generating dense content.
- Findings: Logic and aesthetics trade off: richer text descriptions and complex layouts can improve logic while reducing visual quality, and test-time scaling targets both abilities.The benchmark evaluates open- and closed-source unified models and text-to-image models with scaling methods.
2 Related Work
Prior benchmarks evaluate general text-to-image capabilities, while scientific-figure systems increasingly automate figure creation through extraction, layout, agentic rendering, or code-based generation. These approaches motivate specialized evaluation of academic illustrations.
- Text-to-Image Generation Benchmarks: GenEval tests compositional prompt following, WISE targets knowledge-intensive factual consistency, and T2I-CompBench++ evaluates compositional reasoning with tailored automatic metrics.These benchmarks cover complementary capabilities in general-purpose text-to-image generation.
- Research Gap: Academic figure benchmarks extend general text-to-image evaluation toward the specialized demands of methodology-diagram generation.This shift reflects the need to assess scientific figures beyond broad compositional or factual consistency.
- Automated Scientific Figure Generation: Scientific figure generation has progressed from extract-and-layout pipelines toward agentic, code-based rendering for more controllable outputs.PPSGen, PPTAGENT, and Paper2Poster exemplify these directions, although they largely reorganize or stylize source assets.
3 AIBench
AIBench separates objective logical accuracy from subjective aesthetics, using structured, multi-level VQA to evaluate whether generated academic illustrations align with their source methods. It combines automated filtering and human review into a curated benchmark and evaluates models across fine-grained and global reasoning dimensions.
- Evaluation motivation: AIBench decouples academic-illustration evaluation into objective logic assessment with VQA and subjective aesthetic assessment with a model-based evaluator.This separation addresses the metric ambiguity of using one VLM to score both logical errors and aesthetic flaws.
- 3.1 Data Collection & Curation: The benchmark sources recent papers from CVPR, ICCV, ICLR, and NeurIPS, extracting methodology text and primary architecture figures through automated parsing, ranking, selection, and verification.Uniform sampling across conferences produces approximately 3,000 text–figure pairs before later curation.
- 3.2 QA Construction: AIBench constructs a logic-directed graph G = (V, E, P) from methodology text, preserving source terminology while representing components, data flows, and architectural phases.The graph converts lengthy, weakly structured method descriptions into an explicit intermediate representation for QA synthesis.
- 3.2 QA Construction: Multi-level QA pairs probe component existence, local topology, phase architecture, and global semantics, progressing from node-level checks to end-to-end design intent.These levels also test text rendering, local layout clarity, macro-layout coherence, and global visual clarity.
- 3.5 Evaluation Protocol: For logic evaluation, a multimodal solver answers each question using only the generated image, while question-level global averaging gives every visual reasoning judgment equal weight.The framework decomposes complex global logic into atomic QA pairs, reducing the solver’s reasoning burden and producing more stable estimates for complex illustrations.
4 Experiments
Experiments show a large hierarchy among academic-illustration models, with closed-source systems stronger on dense logical and structural generation. Results also expose a logic–aesthetics trade-off, recurring visual reasoning failures, and gains from test-time scaling and robust VQA evaluation.
- Overall Performance and Model Hierarchy: Nano Banana Pro achieves 77.77 overall, while closed-source models substantially outperform open-source models on dense logical reasoning and structural generation.The benchmark reveals a larger capability gap than general generative benchmarks, where open- and closed-source performance often appears saturated.
- The Trade-off Between Logical Fidelity and Aesthetics: Generated illustrations with higher structural information typically achieve better logical fidelity but lower aesthetic quality.GPT-Image-1.5 has the strongest aesthetics among closed-source models, whereas Nano Banana Pro provides higher information density with acceptable aesthetics.
- Comparison with Original Images: Nano Banana Pro (77.77) and Seedream 5.0 (73.23) outperform the Original Image baseline (70.09), mainly in Component completeness and Global Semantic alignment.The paper attributes this counterintuitive result mainly to human authors omitting components that models explicitly align with methodology text.
- Typical Failure Modes in Logical Rendering: Current models fail through missing components, layout errors, hallucinated reasoning, and unclear text rendering when translating complex methods into visual layouts.Examples include omitted functional pathways, incorrect spatial arrangements, disconnected data flows, and obscured architectural stages.
- Test-Time Scaling: Test-time scaling improves performance by separately addressing long-context reasoning and imperfect visual rendering.Rewriting raises Qwen-Image-2512 from 42.83 to 58.39, while Best-of-N raises Wan2.6 from 65.84 to 67.23.
- Robustness Analysis: AIBench’s VQA-based scores vary slightly across VLM solvers, but evaluated T2I model rankings remain remarkably consistent.This indicates that benchmark conclusions are robust to the choice of VLM QA solver.
5 Conclusion
The paper introduces AIBench as a fine-grained VQA-based benchmark for complex academic illustrations and evaluates model capabilities in logical reasoning and high-density generation. It also constructs logic graphs and multi-level QA pairs while documenting capability gaps and test-time scaling effects.
- Experiments reveal capability gaps for long, complex reasoning and high-density generation, while test-time scaling can raise current performance ceilings.
- The benchmark constructs Text-to-Logic Directed Graphs by extracting core logic from method text and generating graphs from execution logs.
- Multi-level QA generation covers component existence, local topology, phase architecture, and global semantics.
- The dataset’s ground truth was manually reviewed by domain-expert Ph.D. students across 300 annotated articles.
B.1 More Experimental Settings and Details
The appendix details model-specific prompting, graph representations, resolution settings, and a unified VQA evaluation protocol. It also describes the extended test-time-scaling evaluation presented in Table A1.
- Model Inference and Prompt Design: Closed- and open-source models receive separate inference prompts tailored to their instruction-following and structural-understanding capabilities.
- Text-to-Logic Directed Graph: Rendered logic graphs use purple rectangular boxes for processing modules, elliptical boxes for data entities, black arrows for data flow, and yellow boxes for phases.
- Resolution Settings: All generated illustrations use each model’s default resolution with a minimum output size of 1024×1024 pixels.
- QA Evaluation Protocol: The VQA protocol applies one straightforward QA template to automatically generated and human-checked multi-level question-answer pairs.
- Test-Time Scaling: Table A1 evaluates explicit reasoning enhancements, including text rewriting and SVG-based structural priors, across models with different capabilities.
B.2 More Analysis on Test-Time Scaling
Additional experiments show that test-time-scaling strategies depend strongly on the model’s native comprehension and generation capacity. SVG priors can help capable models but severely harm weaker ones, while gains saturate for exceptionally strong models.
- Test-time-scaling experiments compare text rewriting and intermediate SVG structural priors across models with different capabilities.
- Rewriting Mechanisms: Rewriting mechanisms affect advanced closed-source models differently because some already include optimized internal prompt enhancement.
- Structural Priors: Qwen-Image-2512’s score collapses from 42.83 to 12.20 when SVG code is injected as a structural prior.
- Structural Priors: Nano Banana Pro improves only from 77.77 to 78.03, as the intermediate Text →SVG Code →Image process may bottleneck an already capable end-to-end planner.
B.3 The Trade-off Between Logical Fidelity and Aesthetics
Additional qualitative examples support a trade-off between logical fidelity and aesthetic quality in academic illustration generation. Seedream 5.0 produces clearer logical organization, but the passage indicates that this improvement is accompanied by a cost to aesthetics.
- Seedream 5.0 produces illustrations with clearer logical organization and more accurately represented steps and progression.
- The improved logical fidelity is accompanied by a decrease in aesthetic quality.
B.4 Academic Illustrations Generated by Different Models on AIBench
AIBench compares academic illustrations generated from the same method text and evaluates their structural and visual quality through targeted questions. Qualitative examples show differences among model types and illustrate how each case links method text, a pipeline image, and QA answers.
- Closed-source models generally outperform open-source and unified models in logical consistency and visual quality.
- Nano Banana Pro produces illustrations strong in both structure and aesthetics, while some unified models generate semantically meaningless illustrations.Emu3.5 often includes substantial method text, which may partly explain its relatively higher AIBench scores than other unified models.
- Figures A5–A7 provide qualitative comparisons of outputs from closed-source, open-source, and unified models using the same method text.
- Each AIBench case pairs method text with a generated pipeline image and a curated QA list containing evaluation questions and answers grounded in the method and pipeline.
- The QA list targets deep structural reasoning by asking models to identify components, trace directional information flow, and infer relationships between modules.Each question has a precise answer derived from the generated image.
C Limitation and Future Work
AIBench is currently curated from premier artificial intelligence conferences, limiting representation of scientific fields with different diagrammatic conventions. Future work aims to broaden disciplinary coverage and improve reasoning, rendering, structural generation, and aesthetic quality.
- AIBench lacks representation from biology, chemistry, and materials science, whose diagrams may use different conventions.Examples include molecular structures and complex experimental apparatuses.
- Future expansion will cover more scientific fields to evaluate the cross-disciplinary generalization of generative models.
- Future research should improve reasoning over long, complex contexts, text rendering, structural generation, and aesthetic quality.
- Suggested directions include architectural improvements, tailored test-time scaling, and reinforcement learning to optimize logical accuracy and aesthetic quality.
D Copyright and Licensing
The benchmark uses publicly accessible arXiv source papers under open-access licenses and frames its methodology extraction and diagram use as non-commercial academic research. Its construction includes prompts for converting method text into logic graphs and generating multilevel structural questions.
- The source papers are publicly accessible through arXiv under open-access licenses such as CC BY 4.0.
- The benchmark limits use of methodology descriptions and diagrams to non-commercial academic research under fair-use and academic-exception principles.
- The Text-to-Logic Directed Graph prompt extracts an execution pipeline by separating logical steps from data flows and ignoring descriptive noise.
- The extraction prompt excludes comparisons, qualitative claims, and irrelevant implementation details while preserving original component and data-structure names.
- QA-construction prompts define four levels spanning component existence, local topology, phase architecture, and global semantics.
- The Level 2 prompt generates local-topology questions from a full logic graph while testing underlying architecture rather than reference-image styling.