PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility

Honglin Lin, Chonghan Qin, Zheng Liu, Qizhi Pei, Yu Li, Zhanping Zhong, Xin Gao, Yanfeng Wang, Conghui He, Lijun Wu

arXiv:2601.17027v1cs.CVcs.AI

TL;DR

Multimodal scientific reasoning lacks rigorously synthesized visual data because T2I models can be visually plausible yet scientifically incorrect. The paper compares pixel-based and programmatic synthesis, introduces ImgCoder and SciGenBench, and finds that verified synthetic images improve downstream reasoning while exposing method-specific limitations.

  • Problem

    Scientific image synthesis remains difficult because T2I models often violate the geometric, physical, and relational constraints required for scientifically rigorous visuals.

  • Method

    The paper compares pixel-based and programmatic generation, proposes ImgCoder’s Understand→Plan→Code workflow, and evaluates images with SciGenBench and inverse validation.

  • Results

    Code-based ImgCoder achieves the top Rinv of 77.87%, while verified synthetic images consistently improve downstream multimodal scientific reasoning.

  • Takeaways & Limitations

    High-fidelity, logic-grounded scientific image synthesis is a viable scalable pathway for advancing multimodal scientific reasoning.

  • Takeaways & Limitations

    Inverse validation uses Gemini-3-Flash as the VQA engine to reduce failures attributable to solver limitations.

Abstract

from arXiv · show

While synthetic data has proven effective for improving scientific reasoning in the text domain, multimodal reasoning remains constrained by the difficulty of synthesizing scientifically rigorous images. Existing Text-to-Image (T2I) models often produce outputs that are visually plausible yet scientifically incorrect, resulting in a persistent visual-logic divergence that limits their value for downstream reasoning. Motivated by recent advances in next-generation T2I models, we conduct a systematic study of scientific image synthesis across generation paradigms, evaluation, and downstream use. We analyze both direct pixel-based generation and programmatic synthesis, and propose ImgCoder, a logic-driven framework that follows an explicit "understand - plan - code" workflow to improve structural precision. To rigorously assess scientific correctness, we introduce SciGenBench, which evaluates generated images based on information utility and logical validity. Our evaluation reveals systematic failure modes in pixel-based models and highlights a fundamental expressiveness-precision trade-off. Finally, we show that fine-tuning Large Multimodal Models (LMMs) on rigorously verified synthetic scientific images yields consistent reasoning gains, with potential scaling trends analogous to the text domain, validating high-fidelity scientific synthesis as a viable path to unlocking massive multimodal reasoning capabilities.

1 Introduction

Scientific image synthesis is a bottleneck for multimodal scientific reasoning because current T2I systems often produce visually plausible images that violate strict scientific constraints. The paper studies generation paradigms, benchmarking, and downstream utility, introducing ImgCoder and SciGenBench.

  • Scientific visuals must satisfy strict geometric, physical, and relational constraints that existing T2I models often fail to enforce.
  • Next-generation T2I models motivate a systematic investigation into whether improved semantic understanding and visual control can support scalable multimodal data generation.
  • ImgCoder uses a logic-driven Understand→Plan→Code workflow that decouples reasoning from rendering and exposes a precision–expressiveness trade-off between programmatic and pixel-based synthesis.
  • SciGenBench contains 1.4K problems across 5 domains and 25 image types, evaluating information utility and logical validity through LLM judging and inverse validation.
  • Fine-tuning LMMs on scientifically verified synthetic image–text pairs consistently improves scientific reasoning, with potential scaling trends similar to text-only settings.

2 Related Work

Prior text-to-image work emphasizes visual fidelity, while scientific image synthesis requires precise structures governed by domain laws. The paper frames this gap through pixel-based and code-based generation and presents a framework combining synthesis, benchmarking, and evaluation.

  • Modern diffusion and autoregressive T2I systems achieve high visual fidelity, but structured scientific diagrams remain difficult to generate correctly in pixel space.
  • Code-based methods improve structural control through executable visualization specifications, although manually defined templates can restrict output diversity.
  • Earlier benchmarks rely mainly on visual fidelity and semantic-alignment metrics, while newer reasoning-informed evaluations assess plausibility or contextual consistency.
  • Scientific synthesis differs from natural-scene generation because it must materialize abstract axioms into precise visual structures under strict domain laws.
  • The methodological overview combines ImgCoder generation, SciGenBench construction, and a multi-faceted evaluation framework.

3 Scientific Image Generation

The paper compares direct pixel-based synthesis with executable code-driven generation for scientifically constrained images. ImgCoder adds explicit planning before code generation to improve structural execution and logical fidelity.

  • Pixel-based models translate text directly into image space, whereas code-driven approaches produce executable specifications that are deterministically rendered.
  • Scientific image generation is formulated as conditional generation constrained by latent scientific axioms, with the objective tied to downstream reasoning outcomes.
  • Direct synthesis uses proprietary and open-source T2I models together with constraint injection for information fidelity, negative constraints, and textbook-style presentation.
  • ImgCoder follows an explicit Understand→Plan→Code workflow to generate executable code for deterministic rendering.
  • A reasoning and planning stage defines visualization intent before code synthesis, improving compilation success and logical fidelity in complex illustrations.

4 SciGenBench: Benchmarking Scientific Image Synthesis

SciGenBench evaluates scientific image synthesis through curated, hierarchically organized data and a hybrid protocol focused on information utility and logical correctness. Its components combine atomic visual quizzes, LMM judging, inverse validation, conventional metrics, and downstream training utility.

  • SciGenBench benchmarks scientific images using curated data construction, hierarchical taxonomy, and hybrid evaluation of information utility and logical correctness.
  • Data Acquisition & Selection: The benchmark combines an instruction-driven set for synthesis assessment with a real-world visual reference set for distributional comparison.
  • Data Acquisition & Selection: Its data pipeline filters scientifically rigorous sources for visualizability, classifies them across 5 subjects and 25 image types, and constructs atomic quizzes from verifiable facts.
  • Data Acquisition & Selection: Blind filtration discards questions solvable from text alone, while density-based selection and expert review prioritize information-rich, logically consistent samples.
  • Evaluation Framework: The evaluation combines LMM-as-Judge scoring, inverse validation, reference-based metrics, and downstream LMM fine-tuning to assess logical correctness and functional utility.
  • Evaluation Framework: Inverse validation measures the proportion of images whose associated atomic quizzes are all answered correctly, using a VQA engine to reduce solver-related validation failures.

5 Benchmarking Generative Capabilities

Benchmarking shows that scientific image synthesis depends on structural and logical correctness, not visual realism alone. Code-driven ImgCoder generally improves precision, while pixel-based models retain advantages in expressive rendering and some visually rich domains.

  • Benchmark setup: Current benchmarking compares generative models and programmatic frameworks using inverse validation and LMM-as-Judge evaluations.SciGenBench reports Rinv and Judge scores, while standard metrics are computed on the real-image SeePhys subset.
  • Quantitative results: Nanobanana-Pro reaches 73.41% Rinv, while HunyuanImage-3.0 and Qwen-Image remain below 40% Rinv and under 0.8 on structure-sensitive Judge dimensions.The closed-source advantage appears in both inverse validation and LMM-as-Judge evaluations.
  • Quantitative results: Gemini-3-Pro-ImgCoder achieves the top 77.87% Rinv, while Qwen3-ImgCoder improves over Qwen-Image from 38.86% to 56.38% Rinv.The results attribute these gains to code-based execution rather than closed data alone.
  • Evaluation limitations: Perceptual metrics such as PSNR and FID do not reliably track inverse validation or reasoning-oriented Judge scores.Perceptual fidelity aligns more with E&R than with the stricter C&F and L&P dimensions, so visual similarity can mask factual or relational errors.
  • Qualitative failure analysis: Structural and dense-data errors remain persistent bottlenecks even when traditional compositional and rendering errors are reduced.ImgCoder performs better in high-precision regimes because its code-driven process supports rigorous structural control.
  • Precision–expressiveness trade-off: Pixel-based models offer richer visual expressiveness for physical scenes, while code-based outputs are often schematic or flat.The spring-system comparison illustrates the complementary strengths of the two paradigms.
  • Precision–expressiveness trade-off: Pixel-based models can produce visually smooth but mathematically inaccurate plots, whereas code-based execution preserves exact geometric and data-coordinate relationships.For y = x ln x, Nanobanana-Pro misses correct intercepts and extrema; code-based methods ensure exactness through deterministic execution.
  • Domain-specific results: Domain performance depends on structural constraints and information density: ImgCoder leads in mathematics, physics, and universal diagrams, while pixel-based models favor visually rich biology and chemistry cases.Chemistry is mixed, with molecular structures favoring code-based reasoning and crystal or reaction diagrams favoring pixel-based models.

6 Data Utility for Downstream Reasoning

The paper evaluates whether synthetic scientific images improve downstream multimodal reasoning and finds consistent gains from higher-quality, filtered data, with accuracy increasing as dataset scale grows.

  • Synthetic images improve downstream multimodal reasoning across model variants on GEO3K and MathVision.The evaluation compares training with synthetic images against a baseline.
  • 58.2 average score from Nanobanana-Pro (Filt) represents a 3.7-point gain over the 54.5 baseline.
  • ImgCoder variants show a clear advantage on the more challenging MV benchmark compared with pixel-based counterparts.
  • Higher-quality images and filtration improve downstream performance, indicating sensitivity to image quality rather than data quantity alone.Nanobanana-Pro data outperforms Qwen-Image, 58.1 vs. 57.1.
  • Accuracy rises from 43.9% to 46.1% (+2.2%) as synthetic training data scales from 50 to 1.4K samples.The performance curve follows a log-linear trend without observable saturation.

7 Conclusion

The paper frames scientific image synthesis as a precision–expressiveness trade-off between code-based and pixel-based generation. It concludes that logic-grounded, high-fidelity synthetic images can support multimodal reasoning.

  • Scientific image synthesis exhibits a precision–expressiveness trade-off between pixel-based and code-based generation.
  • SciGenBench evaluates structural correctness and information utility in generated scientific images.
  • ImgCoder improves structural precision and reduces failure modes, while pixel-based models provide richer visual expressiveness.
  • High-fidelity synthetic images consistently improve downstream multimodal reasoning performance.

A Experimental Details

The experiments cover representative open-source and proprietary T2I models alongside multiple ImgCoder backbones. Inference uses default settings, with explicit robustness measures for code execution failures.

  • The evaluation includes HunyuanImage-3.0 and Qwen-Image as open-source T2I models.
  • Proprietary evaluations include GPT-Image-1, GPT-Image-1.5, Seedream-4.0, Flux2-flex, Nanobanana, and Nanobanana-Pro.
  • ImgCoder is evaluated with Qwen3, Gemini-3-Flash, and Gemini-3-Pro backbones.
  • Open-source models run locally on NVIDIA A100 GPUs, while closed-source models use official APIs and official prompt rewriting for open-source inference.
  • ImgCoder retries code extraction or compilation failures up to three times before returning a blank image.

A.3 Downstream Training Configuration

Downstream training uses filtered and unfiltered synthetic datasets to fine-tune Qwen3-VL-8B-Instruct, with fixed rollout and evaluation settings based on VeRL and Compass-Verifier-8B.

  • Four synthetic datasets support reinforcement learning fine-tuning, with filtered Nanobanana-Pro and Qwen-Image subsets constructed by removing incorrect images.
  • Training runs for 200 global steps with batch size 128, eight rollouts per prompt, and maximum response length 8192 tokens.
  • MathVisionmini and Geometry3Ktest provide evaluation benchmarks, with MathVisionmini reported using AVG@4 for stability.
  • Compass-Verifier-8B serves as the judge model for reward computation during training and accuracy evaluation.

B.1 Distributional Gap with Real Images

Across evaluated models, synthetic scientific images occupy a distribution distinct from real scientific images in both representation space and frequency content. This gap reflects a broadly shared synthetic visual style and spectral bias rather than an issue isolated to one model.

  • All 11 evaluated models place most synthetic images away from the tightly clustered real-image manifold in CLIP embedding space.The analysis compares every evaluated model against the real SeePhys dataset.
  • The shared representation gap indicates a consistent “digital scientific style” that differs visually from real scientific literature.Synthetic distributions are internally consistent but remain distinct from the natural distribution of real-world scientific images.
  • Every generated model exhibits higher high-frequency spectral energy than real scientific images.The real images form the lower-energy reference curve in the frequency-domain analysis.
  • The spectral gap is attributed to synthetic images lacking natural degradation processes such as printing imperfections, scanning noise, and paper texture.

B.2 Domain-specific Breakdown

Performance varies by scientific domain: code-based generation is strongest when images require strict structure and information density, whereas pixel-based models remain competitive in visually organic or expressive settings. This pattern motivates complementary or hybrid synthesis strategies.

  • Overall pattern: Code-based methods consistently dominate structure-intensive domains, while pixel-based models retain advantages in visually expressive or loosely constrained scenarios.
  • Mathematics and Physics: 69.86% and 75.39% inverse validation rates are achieved by Gemini-3-Pro-ImgCoder in Math and Physics, respectively.Its corresponding LMM-as-Judge scores are 1.94 in Math and 1.93 in Physics.
  • Chemistry: Chemistry shows a mixed pattern: code-based models remain competitive on abstract symbolic tasks, while pixel-based models perform comparably or slightly better on visually rich subdomains.The contrast is between molecular structures and electron configurations versus crystal structures and reaction schematics.
  • Biology: 74.51% Biology Rinv and a 1.89 Judge score are attained by Nanobanana-Pro, the strongest reported Biology performance.Cell diagrams and molecular processes benefit from textures, organic shapes, and visual priors from natural imagery.
  • Universal Diagram Types: 72.85% Universal performance and a 1.91 Judge score are achieved by Gemini-3-Pro-ImgCoder on cross-domain diagram types.Its advantage is linked to precise alignment, spacing, and symbolic accuracy in tables, grids, and charts.
  • Summary: Domain performance is primarily governed by structural constraint and information density, producing complementarity between code-based and pixel-based generation.Code-based methods excel in Math, Physics, and Universal diagrams, while pixel-based models remain competitive in Biology and some Chemistry subfields.
Loading 2601.17027v1…