PaperlayerView exact revision on arXiv

Source-linked AI summary

SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation

Zhiyuan Ma, Zhengfeng Shi, Yuning An, Peize Li, Jiabao Wei, Ruijie Li, Junhao Xiao, Jianjun Li, Bowen Zhou

arXiv:2606.30124v1cs.CV

TL;DR

Scientific image generation still lacks reliable semantic alignment and reasoning under physical, topological, and causal constraints. SciIR introduces a semiotic-triad dataset and benchmark with reasoning annotations and atomic checks, while fine-tuning on its data raises SciIR-Bench from 35% to 43%.

  • Problem

    Scientific image generation lacks logic-annotated data and fine-grained standards for evaluating physical, topological, and causal correctness.

  • Method

    SciIR organizes scientific reasoning into entity structure, scientific process, and scientific law, providing SciIR-82k with Sci-RCoT annotations and SciIR-Bench with atomic checklists.

  • Results

    Fine-tuning Qwen-Image-2512 on SciIR-82k raises the SciIR-Bench score from 35% to 43%, with largest gains on entity structure and scientific process tracks.

  • Takeaways & Limitations

    SciIR provides a resource for evaluating and training scientific image reasoning beyond perceptual quality, with improvements in scientific consistency after reasoning-dense training.

  • Takeaways & Limitations

    SciIR-82k is biased toward published, standardized figures and underrepresents atypical or unconventional diagrams, while SciIR-Bench emphasizes scientific correctness over visual aesthetics.

Abstract

from arXiv · show

While Text-to-Image (T2I) models have shown remarkable success in generating photorealistic visual content, they still struggle with the rigorous semantic alignment and logical reasoning required for scientific imagery. Inspired by Peirce's Semiotic Triad, we introduce Scientific Image Reasoning (SciIR), a comprehensive resource for training and evaluation of scientific image generation. We formalize scientific reasoning into three core dimensions: Entity Structure (Icon), Scientific Process (Index), and Scientific Law (Symbol). Specifically, to overcome the scarcity of training data in scientific image generation, we elaborately create SciIR-82k, a large-scale dataset containing over 80,000 high-quality scientific image-text pairs from cutting-edge publications. The dataset is hierarchically organized according to the semiotic dimensions and incorporates a Scientific Reasoning Chain-of-Thought (Sci-RCoT) to explicitly model underlying visual logic. For evaluation, we propose SciIR-Bench, which aligns with these three semiotic levels and employs an Atomic Checklist to convert the outcome-oriented scientific accuracy into process-oriented, verifiable, fine-grained questions. Our extensive experiments reveal significant deficiencies in current models' scientific reasoning capabilities. Furthermore, by fine-tuning on the SciIR-82k dataset, we developed the Qwen-Image-SciIR model, which achieves a substantial improvement on the SciIR-Bench, increasing the final score from 35\% to 43\%, laying a solid foundation for future advances in scientific image generation.

1 Introduction

SciIR addresses the gap between visual quality and scientific reasoning robustness in image generation through a semiotic triad-based dataset and benchmark. Its dataset, evaluation framework, and fine-tuned model target scientifically valid visual logic and improve SciIR-Bench performance from 35% to 43%.

  • Motivation: Recent T2I advances improve visual realism, stylistic diversity, semantic alignment, instruction following, and ambiguity resolution, but models still struggle with scientific validity and domain-specific constraints.The introduction emphasizes that perceptual fidelity does not equate to reasoning robustness.
  • Challenges: Scientific image generation is hindered by scarce logic-annotated resources and evaluation frameworks that lack scientific-correctness standards.Existing datasets lack the visual logic needed to learn dependencies mapping text into scientifically accurate structures.
  • Contributions: SciIR-82k contains over 80,000 scientific image-text pairs from Nature and Nature Communications, augmented with Sci-RCoT annotations of latent visual reasoning pathways.The annotations are designed to train models on underlying scientific logic.
  • Contributions: SciIR-Bench organizes evaluation around multidimensional scientific correctness and uses an atomic checklist to produce fine-grained, verifiable questions.The benchmark is introduced as the first systematic categorization of these evaluation tracks.
  • Results: 35% to 43%: Qwen-Image-SciIR raises the final SciIR-Bench score by fine-tuning Qwen-Image-2512 on SciIR-82k.The model is presented as a strong open-source baseline and starting point for future scientific image reasoning research.

2 Related Work

Prior T2I datasets and benchmarks largely emphasize descriptive captions, perceptual quality, prompt alignment, or broad semantic plausibility rather than fine-grained scientific reasoning. SciIR addresses this gap by decomposing scientific correctness into law, structure, and process while diagnosing atomic violations.

  • Scientific T2I Datasets: Synthetic datasets offer large-scale, controllable supervision but often prioritize visual plausibility and stylistic diversity over rigorous logical consistency.They may also inherit biases from their source models.
  • Scientific T2I Datasets: Non-synthetic datasets cover everyday concepts through web image-text pairs, but typically use short captions lacking domain knowledge and explicit logical structures.Scientific datasets such as Science-T2I leverage specialized knowledge to mitigate this limitation.
  • Evaluation Benchmarks: Existing benchmarks progress from perceptual quality to prompt alignment and semantic plausibility, while scientific-figure benchmarks focus on flowcharts, statistical diagrams, or broad diagram interpretation.Representative metrics and frameworks include IS, FID, T2I-CompBench, GenEval++, WISE, T2I-ReasonBench, PaperBananaBench, FigureBench, and SridBench.
  • Evaluation Benchmarks: No existing benchmark comprehensively covers scientific law, entity structure, and scientific process or diagnoses specific atomic violations such as broken causal links.The paper frames this deficiency as structural opacity and argues that scientific correctness requires systematic decomposition rather than a monolithic metric.
  • Scientific Illustration Generation: Automated scientific-illustration methods emphasize publication-ready layout and aesthetics, workflow visualization quality, or programmatic synthesis rather than pixel-level reasoning.AutoFigure, PaperBanana, and ImgCoder represent these respective directions; ImgCoder lacks the visual reasoning described in the passage.

3 SciIR-82k Dataset

SciIR-82k is a large-scale dataset of over 80,000 high-quality scientific image-text pairs, organized around Peirce’s Semiotic Triad and built through automated extraction, filtering, stratification, and reasoning-driven annotation. Its Sci-RCoT pipeline reverse-engineers visual evidence into structured reasoning traces and compact, scientifically grounded prompts.

  • Dataset construction: SciIR-82k contains over 80,000 high-quality scientific image-text pairs with complete annotations, constructed through a multi-stage automated pipeline grounded in the Semiotic Triad.The corpus uses CC BY 4.0 articles from Nature and Nature Communications; about 360k raw figures are decomposed, standardized to 1024×1024, filtered, and manually verified.
  • Semiotic stratification: The taxonomy defines Entity Structure as iconic geometric and spatial fidelity, Scientific Process as indexical causal or temporal relations, and Scientific Law as symbolic adherence to abstract rules.These tracks cover entity reconstruction, state transitions and workflows, and constraints such as conservation of energy and molecular valence.
  • Semiotic stratification: Qwen3-VL evaluates each sample on a relevance score s ∈[1, 10] and routes it to an annotation pipeline according to its dominant semiotic attributes.The stratification aligns raw data with the three reasoning tracks: Entity Structure, Scientific Process, and Scientific Law.
  • Reasoning-driven annotation: The annotation pipeline reverse-engineers images into Sci-RCoT by extracting taxonomy-guided visual evidence, integrating style and text requirements, and distilling the reasoning trace into concise prompts.Qwen3-VL prioritizes Image > Caption > Text, while Qwen3-Max applies Term-Substitution to preserve scientific semantics and explicitly required text.
  • Reasoning-driven annotation: The resulting annotations explicitly align image structures, textual elements, and reasoning units, providing controllable supervision for training and evaluating scientific image generation.The pipeline transforms visual evidence into structured reasoning traces and compact prompt representations, while atomic checklists target EntityStructure, ScientificProcess, and ScientificLaw.

4 SciIR-Bench

SciIR-Bench evaluates whether text-to-image models can faithfully generate structured scientific content, including labeled entities, spatial and topological relations, and multi-stage processes. It uses a curated, taxonomy-organized benchmark and a VLM-driven atomic checklist with strict sample-level scoring to assess scientific correctness and logical consistency.

  • Benchmark Construction: SciIR-Bench contains 800 test instances selected for broad scientific coverage, high term density, and challenging reasoning complexity.High Term Density is defined as more than three terms.
  • Benchmark Construction: The benchmark divides candidates into four evaluation groups of 200 samples, including a holistic track combining Scientific Law, Entity Structure, and Scientific Process.The other groups represent distinct reasoning intersections among the three tracks.
  • Difficulty Stratification: Samples are stratified into Instruction Following and Intrinsic Reasoning settings using term density, pairing dense inputs with detailed Sci-RCoT and sparse inputs with abstract prompts.Detailed Sci-RCoT reduces ambiguity and tests fidelity to fine-grained scientific instructions.
  • Evaluation Protocol: The VLM-driven evaluation pipeline performs ground-truth extraction, atomic questioning, and evidence-based refereeing instead of relying on holistic image-quality metrics.Each scientific term produces a binary validation query, and the referee must locate and describe the corresponding visual evidence before assigning a verdict.
  • Accuracy Score: A sample passes a reasoning category only when every associated atomic check passes, making one failure sufficient to invalidate that sample for the dimension.The resulting sample-level pass rate is intended to ensure that high scores reflect scientific robustness rather than partial success.

5 Experiments and Analyses

Experiments show that scientific image generation remains difficult for open-source models, especially without explicit reasoning guidance, while Qwen-Image-SciIR improves scientific accuracy through chained reasoning and visual synthesis. Its Atomic Checklist also aligns more closely with expert judgments than existing metrics.

  • Implementation and Inference: Qwen-Image-SciIR uses separate reasoning-planning and visual-generation modules, chaining Sci-RCoT inference into image synthesis across 800 SciIR-Bench samples.The test instances were removed from SciIR-82k to prevent data leakage.
  • Model Comparisons: 95% performance by Nano-Banana-pro indicates solvability, but open-source contenders retain a 60% gap, while Flux-Dev scores <10% on strict tracks.These results associate aesthetic-focused optimization with perceptual fidelity that does not ensure scientific logic.
  • Instruction Following vs. Intrinsic Reasoning: FLUX.1-Kontext-Max accuracy drops from 36% to 13% without dense guidance, showing that explicit Sci-RCoT prompting generally outperforms abstract prompting.The analysis attributes this gap to stronger instruction execution than autonomous derivation of scientific constraints.
  • AutoRegressive vs. Diffusion: 35% for Qwen-Image-2512 exceeds 21% for Show-o2-7B by 14%, yet top Text-track scores remain 15% for Diffusion and 8% for AR.Thus, diffusion leads autoregressive architectures overall, while both exhibit severe text-generation weakness.
  • Fine-tuning and Qualitative Analysis: Fine-tuning raises Qwen-Image-SciIR’s Final Score from 35% to 43%, with gains of 16% in Scientific Process and 9% in Entity Structure.Qualitative comparisons show fewer hallucinations, including improved process depiction, topology, morphology, and domain-prior adherence.
  • Metric Validation: Atomic Checklist correlation with experts is r = 0.692, outperforming VQAScore at r = 0.457 on 200 randomly sampled test cases.Three annotators independently blind-scored 50 cases per evaluation group, with final ratings averaged.

6 Conclusion

SciIR narrows the gap between general text-to-image capabilities and the strict constraints of natural science through a dataset and fine-grained benchmark. Its limitations include coverage bias and an emphasis on scientific correctness over visual aesthetics, motivating broader annotations and hybrid methods.

  • Contributions: SciIR-82k contains more than 80k high-quality science image–text pairs with traceable Sci-RCoT reasoning chains.The dataset is designed to support scientific image reasoning under natural-science constraints.
  • Contributions: SciIR-Bench decomposes scientific correctness into fine-grained, verifiable atomic checks, including topology, causality, and conservation.The benchmark converts scientific correctness into process-oriented checks rather than relying only on outcome-level assessment.
  • Limitations and future work: SciIR-82k is biased toward published, standardized figures and underrepresents atypical or unconventional diagrams.This limitation restricts the dataset’s coverage of scientific image styles and forms.
  • Limitations and future work: SciIR-Bench emphasizes scientific correctness over visual aesthetics.The benchmark’s priority creates a limitation in evaluating aesthetic quality.
  • Limitations and future work: Future work should broaden domain and style coverage, add multimodal and cross-lingual annotations, and investigate hybrid training and evaluation approaches.Suggested approaches include symbolic constraints, weak supervision, and adversarial or counterfactual checks.

A Dataset Source, License, and Compliance … A.5 Privacy and De-identification

The dataset uses open-access scientific figures under CC BY 4.0, with article- and figure-level verification, auditable metadata, attribution and change disclosures, and default-deny handling of sensitive content.

  • A Dataset Source, License, and Compliance: The data sources are limited to open-access articles licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).The ingestion pipeline targets high-quality scientific figures from Nature and Nature Communications.
  • A.2 License Verification SOP: The license-verification SOP confirms Open Access and CC BY 4.0 at the article level, then checks captions and credit lines for Third-Party Material.Material with stricter copyright restrictions is excluded.
  • A.3 Metadata Preservation: Each sample preserves a metadata chain containing the source article DOI, direct URL, figure ID, and explicit license name and URL.These fields support dataset auditability.
  • A.4 Release Format and Attribution: The dataset release provides original author attribution and a source link for every sample.These release practices are specified as part of compliance with CC BY 4.0 terms.
  • A.4 Release Format and Attribution: The release explicitly indicates that images were cropped, resized, and standardized.This disclosure identifies the changes applied to the images.
  • A.4 Release Format and Attribution: Captions and structured annotations are released as derived datasets.They accompany the processed image data.
  • A.5 Privacy and De-identification: Despite the typically low privacy risk of scientific figures, the pipeline applies a default-deny policy to sensitive content.Recognizable faces, clinical images, and figures containing potential patient IDs are excluded.
  • A.5 Privacy and De-identification: Dataset statistics report figure proportions across scientific disciplines and term-count distributions across tracks.These statistics are presented in Fig. 6.

B Dataset Construction Pipeline … B.4 Multi-Label Strategy

The dataset construction pipeline combines automated sub-figure extraction, aspect-preserving standardization, dual-stage quality filtering, and soft multi-label assignment. It uses geometric heuristics, VLM decisions with manual checks, and thresholded reasoning-track scores to produce reproducible, high-quality training data.

  • B Dataset Construction Pipeline: The reproducible preprocessing pipeline covers multi-panel splitting, image standardization, and filtration mechanisms.These components form the overall dataset construction workflow.
  • B.1 Multi-Panel Cropping: A YOLO11-Nano-based automated pipeline extracts scientific sub-figures through inference, geometric filtering, and storage.The pipeline is implemented using a fine-tuned model.
  • B.1 Multi-Panel Cropping: Detected regions are discarded when they are below 128 pixels, have aspect ratios outside [0.33, 3.0], or occupy 75%–90% of the figure area.These heuristics remove noise, icons, narrow or flat artifacts, and likely full-figure layout misclassifications.
  • B.2 Image Standardization: Images are converted to sRGB, resized to 1024 × 1024 pixels, content-aware padded, and resampled with Lanczos filtering.Padding uses a dominant edge color when it exceeds 55%; otherwise, it uses the mean edge RGB value.
  • B.3 Dual-Stage Filtering: Dual-stage filtering combines automated VLM screening with manual review to remove low-quality or irrelevant images.This cascade is designed to ensure high data quality.
  • Stage 1: VLM Filtering: InternVL 3.5 classifies images as KEEP or REJECT with reasons, while low-confidence cases are routed to manual review.Photos, screenshots, and pure-text images are examples of rejected content.
  • Stage 2: Manual Spot-Check: A random 10% of KEEP samples is manually reviewed, and batches with False Positive Rate above 5% trigger prompt refinement.The spot-check estimates filtering errors and supports iterative prompt improvement.
  • B.4 Multi-Label Strategy: Soft labels are binarized into a multi-hot encoding, activating each reasoning-track label when its Qwen3-VL relevance score satisfies s ≥ τ with τ = 7.Samples with all track scores below τ are excluded as low-reasoning-content data.

C SciIR-Bench Data Selection … C.3 Adaptive Difficulty Stratification

SciIR-Bench distills SciIR-82k into 800 test instances through hierarchical quality control, semiotic grouping, and adaptive stratification. The pipeline balances sample complexity, reasoning dimensions, and evaluation of intrinsic reasoning versus instruction following.

  • C SciIR-Bench Data Selection: 800 high-quality test instances are distilled from SciIR-82k through Statistical Quality Control, Semiotic Intersection, and Adaptive Difficulty Stratification.These three dimensions govern the hierarchical selection pipeline.
  • C.1 Statistical Quality Control: Interquartile-range filtering retains samples whose scientific-term counts fall within [Q1, Q3], excluding overly simple or cluttered cases.The filtering also avoids spatial-composition limits that can cause uninformative generation failures.
  • C.1 Statistical Quality Control: All candidates must include valid scientific text-rendering instructions, with secondary IQR filters balancing rendered and retained textual complexity.The filtered lengths are rendered_text_stage2 and retained_text_stage3.
  • C.2 Semiotic Intersection Grouping: Samples are assigned to semiotic groups only when they contain non-empty terms and visualization data in at least two tracks.This criterion targets multimodal constraint handling.
  • C.2 Semiotic Intersection Grouping: The benchmark contains four combinatorial groups with N = 200 each: one All_Three group and three pairwise intersections.The pairwise groups are Entity–Law, Law–Process, and Entity–Process.
  • C.2 Semiotic Intersection Grouping: All_Three samples jointly require Scientific Law, Entity Structure, and Scientific Process adherence.This is the holistic reasoning category.
  • C.3 Adaptive Difficulty Stratification: Within each group, samples below the median term count (< Mterms) receive abstract prompts to test autonomous scientific reasoning.Semantic sparsity requires models to bridge gaps using latent domain knowledge.
  • C.3 Adaptive Difficulty Stratification: Samples at or above the median (≥Mterms) receive detailed Sci-RCoT to evaluate fidelity to fine-grained, multi-step scientific instructions.For complex scenes, Sci-RCoT serves as a dense visual blueprint that helps test omission of critical details.

D Automated Evaluation Protocol … E Experiments

The automated evaluation pipeline combines rule-based atomic checklist generation with Gemini-3-Pro-Preview VQA adjudication, then applies strict category-level veto scoring. Its checklist logic validates text, scientific content, and process-specific visual constraints before binary review verdicts are aggregated into dataset pass rates.

  • D Automated Evaluation Protocol: The reproducible evaluation pipeline has two stages: rule-based checklist generation and VQA-based adjudication, both using gemini-3-pro-preview via the Google API.
  • D.1 Atomic Checklist Generation: Checklist generation converts ground-truth reasoning data into binary validation questions under a strict two-layer validation structure.
  • Generation Logic: Layer 1 checks explicitly required text strings for exact spelling and only explicitly specified positional accuracy, avoiding assumptions from vague prepositions.
  • Generation Logic: Layer 2 decomposes track-specific scientific reasoning into atomic attributes and injects negative constraints for impossible laws, incoherent structures, and broken process flows.
  • D.2 Automated Adjudication: The VLM reviewer receives the generated image, original prompt, and checklist JSON, then evaluates each question through visual evidence retrieval, one-sentence reasoning, and a binary verdict.
  • Reviewer System Prompt: The system requires the reviewer to locate each checklist element visually and justify its Yes-or-No decision using only visual observation.
  • D.3 Strict Scoring Aggregation: A category is valid only when every atomic question passes; one failure, such as a misspelled label or incorrect arrow direction, marks the category sample as failed.
  • E Experiments: The benchmark Pass Rate is the percentage of valid samples across the dataset under this veto-based aggregation.

E.1 Ablation experiments of Qwen-Image-SciIR … G.3 Correlation Coefficients

The paper validates Qwen-Image-SciIR components, benchmark stability, and annotation quality, then specifies a semiotic prompt-and-evaluation pipeline and a human correlation study for Atomic Checklist reliability.

  • E.1 Ablation experiments of Qwen-Image-SciIR: Removing Sci-RCoT, Taxonomy, or Planner degrades Qwen-Image-SciIR’s final score.These three component-level ablations are reported in Tab. 6.
  • E.2 Stability analysis of the benchmark: Gemini-3-Pro exclusively generates and judges checklists, while benchmark rankings and main results remain stable across criteria, judge-model, and wording choices.Qwen3-VL and InternVL3.5 are used only during dataset construction.
  • E.3 Human validation: 450 SciIR-82k samples were human-validated by three domain-expert graduate researchers for visual faithfulness and scientific consistency.The results are reported in Tab. 10.
  • F Prompts: The taxonomy scores ScientificLaw, EntityStructure, and ScientificProcess according to visible relevance, while filtering retains only qualifying scientific illustrations.The scoring task uses a 1–10 relevance scale and forbids speculation beyond clear image evidence.
  • F Prompts: Reasoning extraction produces terms and visualizations from the image and caption, then Sci-RCoT narrates style, required text, entity structure, topology, and processes without unsupported additions.The pipeline prioritizes image evidence over figure title, caption, and article body.
  • F Prompts: Abstract prompt distillation preserves the original visual style while replacing descriptions with reasoning terms, and checklist generation separates text checks from atomic visual checks with track-specific hallucination defenses.ScientificLaw, EntityStructure, and ScientificProcess checks target constraints, topology, and flow or causality, respectively.
  • G.1 Human Study Design: 200 model-generated images were rated blindly by three physics, chemistry, and biology researchers on a 5-point Likert scale, with 50 images per evaluation group.The groups each contained 25 Instruction Following and 25 Intrinsic Reasoning samples, and image ratings were averaged across experts.
  • G. Correlation Analysis: Atomic Checklist correlations are compared with CLIPScore, VQAScore, and VIEScore using averaged human consensus scores, sample-level checklist pass rates, and Pearson’s r, Kendall’s τ, and Spearman’s ρ with two-tailed significance tests.Human consensus ranges from [1, 5], while checklist pass rates range from [0, 1].

G.4 Results Interpretation

The Atomic Checklist aligns more closely with human scientific judgment than general-purpose metrics by evaluating semantic correctness through atomic, evidence-based, track-specific reasoning. It detects subtle structural and causal violations that embedding-based metrics often miss, making it suitable for high-fidelity scientific image evaluation.

  • Metric Alignment: 0.692 Pearson correlation (r), 0.683 Spearman correlation (ρ), and 0.596 Kendall’s tau (τ) demonstrate superior alignment with human judgment.All correlations are statistically significant at p < 0.001.
  • Metric Alignment: CLIPScore (r = 0.345) and VQAScore (r = 0.457) capture surface-level semantics but struggle with subtle structural or causal violations.The passage gives incorrect molecular topology as an example of a missed violation.
  • Atomic Checklist: The Atomic Checklist decomposes scientific correctness into atomic, verifiable questions, enforces evidence-based judgment through visual retrieval, and applies track-specific logic.Its track-specific dimensions are Entity Structure, Scientific Process, and Scientific Law.
  • Evaluation Suitability: Atomic Checklist scores linearly correlate with the rigor of scientific peer review, supporting their use for evaluating high-fidelity scientific image generation.The approach is presented as more suitable than traditional embedding-based metrics for this evaluation setting.
Loading 2606.30124v1…