PaperlayerView exact revision on arXiv

Source-linked AI summary

Science-T2I: Addressing Scientific Illusions in Image Synthesis

Jialuo Li, Wenhao Chai, Xingyu Fu, Haiyang Xu, Saining Xie

arXiv:2504.13129v2cs.CVcs.AIcs.LG

TL;DR

Image generators can create visually plausible images without reliably inferring the scientific principles behind them. Science-T2I, SciScore, and a two-stage alignment framework target that gap. Across 18 models, explicit prompts score roughly 35 points higher than implicit prompts.

  • Problem

    Image-generation training data and standard evaluations rarely test whether models infer correct visual manifestations from scientific concepts.

    Source: §2 · p. 3

  • Method

    We build an expert-annotated adversarial benchmark, develop SciScore from CLIP-H, and align FLUX.1[dev] through supervised and masked reward-based online fine-tuning.

    Source: §1 · pp. 1–2, §5.2 · pp. 9–10

  • Results

    Across 18 models, none exceeds 50/100 on implicit prompts, while explicit prompts score roughly 35 points higher on average.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two-stage framework improves FLUX.1[dev] by over 50% on SciScore and generalizes to complex scenes unseen during training.

    Source: §7 · p. 13

    SciScore failures concentrate in subject-oriented tasks, especially for novel or unseen subjects whose distinctive visual attributes lack sufficient prior exposure.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Current image generation models produce visually compelling but scientifically implausible images, exposing a fundamental gap between visual fidelity and physical realism. In this work, we introduce ScienceT2I, an expert-annotated dataset comprising a training set of over 20k adversarial image pairs and 9k prompts across 16 scientific domains and an isolated test set of 454 challenging prompts. Using this benchmark, we evaluate 18 recent image generation models and find that none scores above 50 out of 100 under implicit scientific prompts, while explicit prompts that directly describe the intended outcome yield scores roughly 35 points higher, confirming that current models can render correct scenes when told what to depict but cannot reason from scientific cues to the correct visual outcome. To address this, we develop SciScore, a reward model fine-tuned from CLIP-H that captures fine-grained scientific phenomena without relying on language-guided inference, surpassing GPT-4o and experienced human evaluators by roughly 5 points. We further propose a two-stage alignment framework combining supervised fine-tuning with masked online fine-tuning to inject scientific knowledge into generative models. Applying this framework to FLUX.1[dev] yields a relative improvement exceeding 50% on SciScore, demonstrating that scientific reasoning in image generation can be substantially improved through targeted data and alignment.

1 Introduction

Image generators can produce polished images without understanding the scientific principles that make scenes realistic. Science-T2I, SciScore, and a two-stage alignment framework target this gap; evaluations expose weak implicit reasoning and show that explicit instructions improve results.

1 Introduction

  • Science-T2I provides over 20k expert-annotated adversarial image pairs and 454 isolated test prompts across 16 scientific domains.Training pairs contrast accurate images with flawed counterparts; domain experts validated both dataset splits.Source: §1 · pp. 1–2
  • None of 18 models scores above 50/100 on implicit prompts, while explicit prompts raise scores by roughly 35 points on average.The contrast indicates that models can render correct scenes when told the intended outcome but cannot reliably infer it from scientific cues.Source: §1 · p. 2
  • SciScore extends CLIP-H with scientific knowledge to assess whether generated images reflect the physical principles implied by a prompt.It captures fine-grained visual cues without language-guided inference and outperforms LMMs such as GPT-4o on this task.Source: §1 · p. 2
  • The proposed two-stage framework applies supervised fine-tuning followed by online fine-tuning with SciScore rewards and subject-based masking.Applied to FLUX.1[dev], the framework improves SciScore by over 50%.Source: §1 · p. 2

2 Science-T2I: Bridging Visual Imagination and Scientific Realism

Science-T2I is designed to test whether image models infer visual consequences from scientific principles, rather than simply render words in a prompt. Its task, prompt, and data designs distinguish scientific reasoning from compositional rendering.

2 Science-T2I: Bridging Visual Imagination and Scientific Realism

  • Science-T2I addresses gaps in training data and evaluation by requiring models to infer visual outcomes from prompts grounded in scientific principles.Unlike datasets centered on textual descriptions or preferences, its prompts test implicit reasoning rather than textual rendering.Source: §2 · p. 3
  • The benchmark spans 16 tasks across physics, chemistry, and biology, each requiring inference of a concept grounded in an underlying scientific principle.Tasks support flexible rephrasing while relying on established scientific principles as consistent ground truth.Source: §2 · p. 3
  • Subject-oriented tasks vary visual features with the subject, whereas condition-oriented tasks vary appearance with an applied condition; SciScore failures concentrate in subject-oriented tasks.Those failures involve subject-specific knowledge rather than generalizable visual patterns.Source: §2 · p. 3
  • Its three prompt types separate implicit scientific inference, direct rendering instructions, and plausible but scientifically incorrect surface interpretations.The implicit prompt tests reasoning, the explicit prompt provides a rendering upper bound, and the superficial prompt supplies a hard negative for preference training.Source: §2 · pp. 3–4
  • The curation pipeline turns task-specific templates into implicit prompts, then expands them into explicit and superficial variants used to generate corresponding images.Domain experts manually verify generated data against established scientific knowledge.Source: Figure 3 · p. 4, §2 · p. 4

3 Benchmarking Scientific Image Synthesis

The benchmark tests implicit scientific reasoning in isolated and complex scenes and compares it with performance under explicit prompts. Results show weak implicit performance across models, while direct descriptions yield substantially higher scores.

3 Benchmarking Scientific Image Synthesis

  • The 454-tuple test set has two non-overlapping 227-tuple splits: simple compositions and complex real-world scenes.The complex split tests scientific accuracy amid distracting contextual elements; strong performance there indicates generalization beyond the training distribution.Source: §3 · p. 4
  • Evaluation scores scene content and scientific reality separately, with the reality score set to zero unless the image achieves a full scene score.Qwen3.5-27B evaluates both scores, and the reality score is normalized to [0, 100].Source: §3 · pp. 4–5
  • Under implicit prompts, the best model scores 47.80/100 and no model exceeds 50; Biology is the hardest domain, with no model above 33%.Most models score between 20 and 35; Z-Image scores 26.73, slightly below FLUX.1[dev] at 26.87.Source: §3 · p. 5, Table 1 · p. 5
  • Explicit prompts yield roughly 35 points higher scores than implicit prompts across all 18 models, showing a persistent composition–reasoning disconnect.Models can depict scientifically correct scenes when explicitly told what to show, but fail to infer the same outcomes from implicit scientific cues.Source: Figure 4 · p. 6, §3 · p. 6

4 SciScore: A Reward Model Grounded in Reality

SciScore adapts CLIP-H to scientific prompt–image preference with complementary text- and image-side objectives, and outperforms multimodal baselines and expert evaluators on both test splits. Its remaining failures concentrate in subject-oriented tasks, while an image-loss weight of λ = 0.25 balances performance across the two splits.

4 SciScore: A Reward Model Grounded in Reality

  • Pretrained CLIP places implicit prompts closer to superficial than explicit counterparts, as surface-level associations outweigh scientific implications.SciScore extends CLIP to assess whether an image reflects the scientific principles implied by a prompt.Source: §4 · p. 6

4.1 Reward Formulation

  • SciScore encodes prompts and images in a shared space and scores their cosine similarity, scaled by a learnable temperature.The text and image representations come from separate transformer encoders.Source: §4.1 · p. 6

4.2 Training Objective

  • IPA aligns implicit prompts toward explicit images, while IEE trains image sensitivity to fine details; λ weights IEE against prompt alignment.The training tuples pair implicit, explicit, and superficial prompts with corresponding explicit and superficial images.Source: §4.2 · p. 7

4.3 Experimental Setup

  • SciScore is evaluated on Science-T2I-S and Science-T2I-C against VLMs, LMMs, and ratings from 10 experts; training takes under an hour on 8 A6000 GPUs.The baseline comparisons include models with and without chain-of-thought prompting.Source: §4.3 · pp. 7–8

4.4 SciScore Outperforms VLMs, LMMs, and Human Evaluators

  • 93.14 on Science-T2I-S and 91.19 on Science-T2I-C, SciScore exceeds human scores (87.01, 86.02) and GPT-4o-mini with CoT (74.97, 77.16).Table 2 reports accuracy (%) on a two-choice selection task across Physics, Chemistry, and Biology.Source: Table 2 · p. 8, §4.4 · p. 8
  • Nearly all failures cluster in subject-oriented tasks requiring subject-specific knowledge, while condition-oriented tasks rely on generalizable visual patterns.Novel subjects may lack sufficient prior exposure to their distinctive visual attributes.Source: §4.4 · p. 8, Figure 5 · p. 9
  • λ = 0.25 achieves the best balance across both splits; removing IEE helps Science-T2I-S but degrades Science-T2I-C.Weights λ ≥ 0.5 shift too much focus to the image encoder at the expense of prompt alignment.Source: §4.4 · p. 8, Table 3 · p. 9

5 Aligning Generative Models with Scientific Principles

The framework first teaches FLUX.1[dev] what scientific images look like, then uses SciScore-guided online fine-tuning to improve reasoning from implicit prompts. The combined approach improves SciScore, while ablations show that both prior SFT and subject-based masking are needed for stable gains.

5 Aligning Generative Models with Scientific Principles

  • SFT exposes FLUX.1[dev] to scientific visual phenomena before OFT optimizes the implicit reasoning ability measured by SciScore.The paper argues preference optimization alone is insufficient when scientific images are absent from pretraining; SFT supplies that missing foundation.Source: §5 · p. 9, §5.1 · p. 9

5.2 Online Fine-tuning with SciScore as Reward Signal

  • OFT uses SciScore rewards to address the remaining gap between learning scientific appearances from explicit pairs and inferring outcomes from implicit prompts.For each prompt, two images are generated and scored; Figure 6 shows GroundingDINO subject masks restricting gradient propagation to relevant image regions.Source: §5.2 · p. 9, Figure 6 · p. 10

5.3 Training and Evaluation Setup

  • Training fine-tunes FLUX.1[dev] on Science-T2I with LoRA SFT for 2,000 steps, then approximately 100 OFT steps using 300 randomly selected implicit prompts; evaluation averages SciScore over five generated images per prompt.Source: §5.3 · p. 10

5.4 Two-Stage Training Yields Over 50% Improvement

  • 23.56 to 28.52 on Science-T2I-S (RI = 53.39%) and 27.26 to 30.11 on Science-T2I-C (RI = 38.31%) with SFT and OFT combined.SFT contributes the larger share; the authors describe it as establishing foundational visual knowledge that OFT then activates from implicit prompts.Source: §5.4 · p. 11, Table 4 · p. 11

5.5 Both Stages and Masking Are Necessary

  • OFT without prior SFT does not improve SciScore, whereas SFT followed by OFT yields a stable increase.The authors attribute the failure to preference comparisons between two scientifically incorrect samples providing insufficient gradient information.Source: §5.5 · p. 11, Figure 7 · p. 11
  • Without masking, standard-rate OFT collapses and halved-rate OFT fails to raise SciScore; masked OFT improves it consistently.The authors attribute the instability to noisy gradients from irrelevant regions, and report that restricting gradients to the scientifically relevant region is essential.Source: Figure 7 · p. 11, §5.5 · p. 12

6 Related Work

Prior work evaluates physical or commonsense reasoning, develops diffusion-model post-training, and measures image quality or compositional fidelity, but leaves gaps in scientific training data and fine-grained scientific evaluation. Science-T2I and SciScore address those gaps with adversarial training data and continuous scoring.

6.1 Scientific Reasoning in Image Generation

  • Existing scientific and commonsense benchmarks provide test prompts but no training data, and use discrete LMM-based scoring that the authors find unreliable for fine-grained judgments.Science-T2I adds adversarial training data, while SciScore provides continuous, calibrated scientific-correctness scores.Source: §6.1 · p. 12, §6.3 · p. 13

6.2 Post-Training Alignment for Diffusion Models

  • Diffusion post-training methods typically assume desired outputs already lie within the pretrained model’s distribution, an assumption the authors say may fail for unseen scientific phenomena.Their framework instead uses SFT to expose the model to scientific visual patterns before online fine-tuning with a domain-specific reward model.Source: §6.2 · p. 12

6.3 Benchmarking Image Generation

  • Most image-generation benchmarks target aesthetics or compositional fidelity rather than scientific correctness; SciScore offers continuous, fine-grained scientific scoring paired with training data.The related-work discussion contrasts this with PhyBench and Commonsense-T2I, which provide test prompts and rely on discrete LMM-based scoring.Source: §6.3 · p. 13

7 Conclusion

The paper reports a gap between visual rendering and scientific reasoning, develops Science-T2I and SciScore, and improves FLUX.1[dev] with a two-stage alignment framework. Its task design is limited to phenomena whose visual outcomes can be expressed unambiguously through text.

7 Conclusion

  • No evaluated model exceeds 50 out of 100 under implicit scientific prompts, while explicit prompts yield roughly 35 points higher scores.The authors conclude that the bottleneck is scientific reasoning rather than visual rendering.Source: §7 · p. 13
  • SciScore, fine-tuned from CLIP-H, surpasses state-of-the-art LMMs and experienced human evaluators by roughly 5 points; the two-stage framework yields over 50% relative improvement on FLUX.1[dev].The fine-tuned model also generalizes to complex scenes not seen during training.Source: §7 · p. 13
  • The authors attribute current scientific-reasoning limitations to missing data and misaligned training objectives, rather than an inherent architectural limitation.They propose Science-T2I and SciScore as resources for research on grounding generative models in reality.Source: §7 · p. 13

A Why We Prioritize Linguistically Describable Tasks

  • Science-T2I excludes reflection and shadow tasks because precise geometric details cannot be specified through text alone and current generators cannot reliably produce controlled pairs.The design principle requires each task’s scientific visual outcome to be expressible unambiguously through an alternative textual description.Source: A Why We Prioritize Linguistically… · p. 19

B Task Definitions and Visual Contrasts

  • The appendix defines visual contrasts for 16 tasks across biology, chemistry, and physics, including conditions such as buoyancy, rust, evaporation, and melting.The tasks are organized as scientific phenomena with visual contrasts for models to reason about.Source: B Task Definitions and Visual Contrasts · pp. 19–20

C All Tasks Follow a Subject + Condition Structure

Science-T2I decomposes prompts into subjects and conditions, then classifies tasks by where their scientific reasoning lies. Its curation pipeline generates aligned prompt variants, selects photorealistic images, and checks scientifically relevant regions.

C All Tasks Follow a Subject + Condition Structure

  • Subject-oriented tasks depend on intrinsic properties that vary by subject, whereas condition-oriented tasks produce similar visual outcomes across subjects under one condition.Examples include buoyancy for subject-oriented reasoning and gravity for condition-oriented reasoning.Source: C All Tasks Follow a Subject +… · p. 20
  • Ten tasks are condition-oriented and six are subject-oriented; SciScore failures concentrate in subject-oriented tasks, while condition-oriented patterns transfer more easily across subjects.The classification is based on whether reasoning arises from intrinsic subject properties or from the applied condition.Source: C All Tasks Follow a Subject +… · pp. 20–21

D Data Curation Pipeline: Prompts, Images, and Quality Control

E Science-T2I Unifies Training and Dual-Purpose Evaluation

Science-T2I pairs a training resource with two complementary evaluation modes, covering both generation and multimodal-model judgment. Its design differs from prior benchmarks that generally provide test prompts for a single evaluation mode.

E Science-T2I Unifies Training and Dual-Purpose Evaluation

  • Its training set contains over 20k adversarial pairs for reward-model training and generative-model alignment.Source: E Science-T2I Unifies Training and… · p. 22
  • The same test set evaluates generation under implicit prompts and tests LMMs and VLMs in a two-choice image-selection task.Both modes use the same prompts and grading criteria.Source: E Science-T2I Unifies Training and… · p. 22
  • Science-T2I uniquely combines training data, generation benchmarking, and LMM evaluation, whereas existing benchmarks are generally test-only and single-task.The comparison is summarized in Table 5.Source: Table 5 · p. 22

F Training Settings for SciScore

G Baseline Setup and Evaluation Process for SciScore

SciScore is evaluated against vision-language models, large multimodal models, and human evaluators using image pairs judged against implicit prompts. Its per-category results include perfect accuracy on multiple tasks, with failures concentrated in subject-oriented tasks.

G Baseline Setup and Evaluation Process for SciScore

  • Each evaluation presents an implicit prompt with images generated from explicit and superficial prompts, and asks which better matches the implicit prompt.Source: G Baseline Setup and Evaluation… · p. 23
  • Baselines include CLIP-H, BLIPScore, SigLIP, four LMMs with GPT-4o-mini tested with and without CoT, and 10 science- or engineering-trained human evaluators.Human evaluators rely on their own knowledge rather than the curators’ scientific-literature checks.Source: G Baseline Setup and Evaluation… · p. 23

H Per-Category Results of SciScore

I Analysis of VLM, LMM, and Human Performance

SciScore separates scientifically correct from incorrect images more reliably than the evaluated VLMs, while LMMs also struggle with visual scientific reasoning. Analyses cover SciScore’s fine-grained visual discrimination, training objectives and settings, and OFT with alternative rewards.

I Analysis of VLM, LMM, and Human Performance

  • SciScore achieves AUC 0.96, compared with 0.56 for CLIP-H, 0.57 for BLIPScore, and 0.60 for SigLIP; evaluated LMMs also struggle despite CoT prompting.The analysis attributes VLM errors to surface-level text-image matching and LMM errors to visual misinterpretation or contradictory reasoning.Source: Figure 14 · p. 24, I Analysis of VLM, LMM, and Human… · pp. 24–25
  • SciScore’s roughly 5% advantage over human evaluators is attributed to task-specific training and consistent application across domains, unlike evaluators’ bounded backgrounds.Source: I Analysis of VLM, LMM, and Human… · p. 25

J Qualitative Analysis of IEE

K Details of Two-Stage Training

L Two-Stage Training Settings

M Additional Results of Online Fine-tuning

N Limitations and Future Directions

The alignment framework’s gains primarily concern 16 predefined tasks, and extending it to new scientific domains requires additional domain-specific training data. Subject-oriented tasks also remain difficult when the model encounters novel or unseen subjects.

N Limitations and Future Directions

  • Performance improves primarily on the 16 predefined tasks; extending alignment to new scientific domains requires additional domain-specific training data.Source: N Limitations and Future Directions · p. 28
  • Novel or unseen subjects remain difficult because the model lacks prior exposure to their specific properties; broader subject coverage or LLM world knowledge are proposed directions.Source: N Limitations and Future Directions · p. 28
Loading 2504.13129v2…