PaperlayerView exact revision on arXiv

Source-linked AI summary

SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

Ye Shen, Yuting Zheng, Dun Pei, Zijian Chen, Wenlong Zhang, Qi Jia, Guangtao Zhai

arXiv:2608.25973v1cs.AIcs.LG

TL;DR

Scientific instruction following remains underexplored because domain-specific requirements and scientific correctness are not captured by existing evaluations. SciMIF builds an expert-derived, discipline-aware benchmark and constraint-injection framework, finding substantial domain variation and persistent difficulty with fine-grained adherence. The results support more application-oriented evaluation and tool-assisted handling of symbolic and discrete constraints.

  • Problem

    Existing evaluations do not adequately test whether MLLMs follow scientific instructions whose constraints depend on disciplinary knowledge, while scientific correctness alone does not establish instruction adherence.

  • Method

    SciMIF analyzes 22 tasks across five disciplines, derives ten constraint groups, and injects compatible scientific and general requirements into existing datasets without changing reference answers.

  • Results

    MLLM performance varies substantially across scientific domains, with chemistry posing greater challenges; models remain vulnerable to fine-grained constraints, and scaling parameters does not consistently improve adherence.

  • Takeaways & Limitations

    Scientific instruction following should be evaluated as a capability distinct from scientific correctness, with future work targeting domain-aware alignment and structure-aware or tool-assisted constraint handling.

  • Takeaways & Limitations

    The paper identifies symbolic and discrete constraints as limitations motivating external numerical computation tools and symbolic engines.

Abstract

from arXiv · show

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .

1 Introduction

SciMIF addresses the underexplored problem of reliable scientific instruction following, where domain-specific constraints and scientific correctness cannot be reduced to surface formatting. It introduces an expert-derived benchmark and scalable constraint-injection framework, then finds substantial disciplinary variation and persistent weaknesses in fine-grained adherence.

  • Motivation: Scientific instructions require domain knowledge because constraints involving entities such as chemical bonds or amino-acid residues cannot be handled through surface text manipulation alone.Their semantics also vary substantially across disciplines.
  • Motivation: Scientific correctness and instruction adherence are distinct capabilities, since valid answers can violate requested formats while compliant outputs can use scientifically invalid reasoning.Separating them helps attribute failures to scientific reasoning, instruction following, or both.
  • Benchmark design: SciMIF covers five scientific disciplines and derives ten functional constraint groups from expert analysis of 22 scientific tasks.The groups capture shared capabilities while adapting concrete constraints to disciplinary knowledge and practices.
  • Benchmark design: The framework identifies implicit requirements, injects compatible scientific and general constraints without changing reference answers, and verifies consistency automatically and manually.The resulting benchmark contains 2,527 samples for evaluating constraint satisfaction, complete instruction fulfillment, and fine-grained adherence.
  • Findings: Experiments show substantial disciplinary variation, with chemistry and geography more challenging, while larger models do not consistently improve constraint adherence.Models also perform worse on general and fine-grained constraints, and scientific correctness remains weakly coupled with adherence.

2 Related Work

Existing scientific benchmarks increasingly target specialized reasoning, while general instruction-following benchmarks assess linguistic, formatting, and structural constraints. SciMIF complements these efforts by deriving discipline-aware constraints from scientific tasks and organizing evaluation across domains and functional capabilities.

  • Scientific benchmarks: Scientific benchmarks have progressed from broad school-level science questions toward expert-level and process-guided reasoning tasks.Their constraints are not systematically derived from the conventions and operational requirements of individual disciplines.
  • Instruction-following benchmarks: General instruction-following benchmarks evaluate explicit linguistic, formatting, and structural requirements, including multimodal inputs.They do not primarily address scientific, domain-specific constraints.
  • SciMIF: SciMIF uses an expert-derived taxonomy from 22 tasks across five disciplines, covering ten functional groups and discipline-adapted constraints.Its organization supports analysis from both disciplinary and capability-oriented perspectives.

3 SciMIF

SciMIF organizes scientific and general instruction requirements into a hierarchical taxonomy and uses it to construct a verified multimodal benchmark from existing scientific datasets. The resulting dataset spans five disciplines, ten functional constraint groups, and 2,527 samples.

  • 3.1 Taxonomy of Scientific Constraints: The taxonomy separates constraints by scientific domain and functional group, preserving domain-specific realizations within shared functional categories.It covers one general domain and five scientific domains, with each constraint characterized by both domain and group.
  • 3.1 Taxonomy of Scientific Constraints: The ten functional groups include procedure, number, method, unit, format, precision, letter, structure, and selection requirements.The supplied taxonomy passage lists these output and task-execution requirements; the displayed list is truncated before the tenth group.
  • 3.2 Data Construction: SciMIF constructs samples through seed preparation, constraint recognition and selection, constraint injection, automatic validation, and human verification.The pipeline begins with questions, optional visual inputs, reference answers, and task types, then checks logical coherence and constraint fidelity before retaining samples.
  • 3.2 Data Construction: Automatic validation checks that injected constraints are included while the ground-truth answer remains unchanged, retrying failed injections up to k times.The validation is represented by the intersection of inclusion and unchanged-answer checks; alternative constraints from the same category are used for retries.
  • 3.3 Data Statistics: The benchmark is built from 13 existing scientific datasets covering 22 task types across chemistry, geography, biology, materials science, and physics.The final dataset contains 2,527 verified samples, and 27.50% include multimodal inputs with associated images.

4 Experiment

The experiments evaluate a diverse set of closed-source and open-source models using complementary metrics for constraint-level, instruction-level, and decomposed-requirement compliance.

  • 4 Experiment: The evaluation includes state-of-the-art closed-source and open-source models, including GPT-5.2, Grok-4-Fast, Gemini-3.1-Pro-Preview, Claude-Sonnet-4.6, and InternVL3.5 series.The supplied experiment passage names representative models from both model categories.
  • 4 Experiment: CSR, ISR, and DRFR provide complementary measures of averaged constraint satisfaction, instruction-level success, and decomposed-requirement compliance.CSR averages constraint-level satisfaction across samples, ISR evaluates whole-instruction success, and DRFR measures compliance across the total number of constraints.
  • 4 Experiment: Model performance is reported across five scientific domains using CSR, ISR, and DRFR to examine disciplinary differences.The domain comparison is summarized in Table 3.

5 Result & Analysis

SciMIF reveals substantial variation in MLLM instruction following across scientific disciplines, constraint domains, model scales, and functional constraint groups. Scientific correctness and instruction adherence also remain difficult to achieve jointly.

  • Performance across disciplines: Biology and materials science achieve the highest instruction-following performance, while chemistry and geography are substantially more challenging.GPT-5.2 reaches ISR scores of 72.82% in biology and 78.07% in materials science; chemistry presents the greatest challenge.
  • Closed-source vs Open-source: GPT-5.2 achieves 65.67% overall, exceeding the 57.39% attained by Qwen3.5-397B-A17B.Closed-source models maintain a consistent performance advantage over open-source models across disciplines, although the gap narrows in physics.
  • Impact of model scale: Increasing model scale does not consistently improve scientific instruction following: InternVL3.5 scores 51.37% at 8B and 51.64% at 38B, while Qwen3.5 falls from 51.37% at 27B to 50.41% at 122B.The results suggest that domain-aware alignment quality is more important than parameter count alone for satisfying complex disciplinary rules.
  • Performance across scientific constraint domains: Chemistry has the lowest average DRFR at 68.2%, whereas physics has the highest at 92.%.Chemistry tasks contain many fine-grained numerical and structural constraints, while physics relies more on coarse-grained methodological and reasoning constraints.
  • General vs scientific constraints: Models perform better on scientific than general constraints; GPT-5.2 scores 88.74% on scientific constraints versus 74.65% on general constraints.Scientific constraints are often coupled to the task, whereas general constraints regulate presentation and organization.
  • Performance across constraint groups: Models struggle most with number, letter, and format groups, with closed-source versus open-source performance gaps reaching 8% to 9%.Number constraints require interpreting scientific structures such as chemical bonds, functional groups, amino-acid residues, and sequence motifs.
  • Relationship between correctness and instruction following: The CF rate remains below 30% across models, while approximately 20% of samples are correct but violate constraints.Several models exceed 30% on incorrect-but-followed samples, showing that format compliance can occur without scientific validity.

6 Conclusion

SciMIF benchmarks multimodal instruction following across five scientific disciplines by modeling shared functional constraints while preserving discipline-specific realizations. Its findings motivate evaluation and training focused on constrained scientific application, including support from numerical and symbolic tools.

  • Conclusion: SciMIF evaluates MLLMs across chemistry, geography, biology, materials science, and physics using ten groups of general and domain-specific constraints.The benchmark assesses integration of disciplinary knowledge with formatting and methodological requirements.
  • Constraint taxonomy: The benchmark preserves discipline-specific meanings and implementations while abstracting constraints into shared functional groups.These variations reflect differences in scientific representations, conventions, reasoning processes, and output requirements.
  • General constraints: General constraints regulate output representation and organization, including format, precision, selection, structure, capitalization, and response count.Examples include returning JSON, specifying decimal precision, and organizing analysis before the final answer.
  • Chemistry: Chemistry constraints cover molecular representations, stoichiometric relationships, reaction processes, and quantities such as atoms, bonds, and functional groups.Terminology constraints may require valid chemical nomenclature or molecular representations such as SMILES.
  • Geography: Geography constraints assess spatial understanding and compliance with conventions involving hierarchies, scales, categories, and structured representations.They can require valid geographical expressions organized according to administrative or spatial hierarchies.
  • Biology: Biology constraints cover entities and processes across molecular sequences and biological systems, including names, lengths, residues, motifs, and procedures.Number constraints may specify protein lengths, sequence lengths, or counts of particular residues or motifs.
  • Materials science: Materials-science constraints address multimodal characterization, property representation, experimental procedures, and structure-property relationships.Format constraints can require properties to be represented as discrete categories, continuous values, or structured labels.
  • Physics: Physics constraints require dimensional consistency, principle-based reasoning, designated laws or equations, and ordered modeling, derivation, calculation, and verification.Unit, method, and procedure constraints explicitly regulate analytical requirements and physically consistent derivations.

A.2 Complete Constraint Specifications

SciMIF defines a structured inventory of general and discipline-specific constraints, then uses it to augment scientific queries while preserving answer validity. The pipeline filters incompatible or redundant requirements, injects constraints sequentially, and verifies each augmentation.

  • Constraint inventory: The inventory organizes 42 constraints into ten functional groups with canonical names, operational descriptions, and evaluation methods.The naming convention distinguishes constraints requiring different disciplinary knowledge or evaluation procedures.
  • Constraint inventory: The specifications cover discipline-specific requirements for chemistry, geography, biology, materials science, and physics, alongside general output constraints.Examples include chemical bonds and reaction procedures, spatial hierarchies, biological sequences, materials characterization, physical laws, formatting, precision, and response structure.
  • Construction pipeline: For each seed sample, the pipeline represents the original query, optional visual input, reference answer, and scientific task type before selecting applicable constraints.Scientific constraints must match the discipline, task type, visual input, and reference answer; general constraints must avoid duplication or conflict.
  • Construction pipeline: Annotators first recognize constraints already stated or implicitly required, preventing redundant injection or conflicts with the original task.The recognized set is obtained through manual recognition and is a subset of the applicable constraint set.
  • Construction pipeline: Selected scientific constraints are adapted to the query, while general constraints are sampled by compatible categories and injected sequentially.At most one constraint is selected from each category, and candidate queries are generated and checked before acceptance.
  • Construction pipeline: Each candidate augmentation is accepted only when the constraint is explicit and the original reference answer remains semantically sufficient.Failed candidates are retried up to k times; if no candidate passes, the category is skipped rather than forcing an invalid augmentation.

B.6 Human Verification

SciMIF applies automatic checks and human review to ensure augmented queries remain coherent, fluent, faithful to their constraints, and answerable by the original reference. Samples that cannot be repaired without changing scientific meaning are discarded.

  • Verification procedure: Automatic validation checks whether injected constraints are explicit and whether the original reference answer remains valid.Human review is added because automatic rules may miss naturalness, implicit contradictions, and ambiguous scientific expressions.
  • Verification criteria: Human verification evaluates logical coherence and fluency, requiring natural integration without contradictions, ambiguity, repetition, or incompatibility.Two annotators review every retained augmented sample against these criteria.
  • Verification criteria: Constraint fidelity requires preserving each constraint’s operational meaning and evaluation condition without altering the scientific question or expected answer.Related terminology alone is insufficient evidence that a constraint has been expressed faithfully.
  • Verification consistency: A 100% agreement rate was observed when two annotators independently calibrated the criteria on 20 randomly sampled instances.This pilot preceded verification of the remaining samples.
  • Verification results: 884 samples were manually revised before inclusion, while unrepaired samples were discarded when repair would change scientific meaning or invalidate constraints.Retained samples preserve the original task, optional visual input, and reference answer.

C Data Sources and Task Coverage

SciMIF draws on 13 existing scientific datasets spanning five disciplines and 22 task types. Its coverage distributes tasks across chemistry, geography, biology, materials science, and physics to increase structural diversity.

  • Data sources: SciMIF is constructed from 13 existing scientific datasets covering chemistry, geography, biology, materials science, and physics.The sources include multiple established datasets within each discipline.
  • Task coverage: The benchmark contains 22 task types with diverse problem formulations, disciplinary conventions, and input modalities.This broad coverage reduces dependence on any single discipline or task formulation.
  • Task coverage: Task counts are eight in Chemistry, four in Geography, three in Biology, four in Material, and three in Physics.The complete task list, source datasets, and sample statistics are provided in Table 12.

D Evaluation Details

SciMIF evaluates constraint satisfaction, complete instruction fulfillment, decomposed requirement compliance, and scientific correctness using programmatic and semantic verification. It also tests whether correctness and instruction following are distinct capabilities.

  • Instruction-following metrics: CSR measures the average proportion of satisfied constraints across all instructions, while ISR measures the proportion satisfying every associated constraint.CSR aggregates constraint-level satisfaction; ISR requires complete instruction-level satisfaction.
  • Instruction-following metrics: DRFR measures requirement-level compliance through scoring questions rather than evaluating each instruction as a single whole.The metric aggregates satisfaction across decomposed requirements.
  • Correctness evaluation: Scientific correctness is evaluated separately with CompassVerifier-32B using the original question, visual input when available, reference answer, and model response.This evaluation does not penalize injected-constraint violations unless they alter the scientific answer.
  • Verification protocols: Two verification protocols are used: scripts handle explicit reproducible criteria, while an LLM judge handles semantic constraint interpretation.Script checks include formats, structures, patterns, and domain-specific validation; the judge assesses reasoning steps and semantic categories.
  • Correctness and adherence: Five of six models showed significant positive correctness–instruction-following associations after Bonferroni correction, while Grok-4-fast did not.Grok-4-fast had p = 0.68 and ϕ = 0.0081; the other significant associations remained modest.
  • Correctness and adherence: ϕ coefficients ranged from 0.0672 to 0.2069, and only 27.04%–36.40% of responses satisfying at least one criterion satisfied both.These results indicate statistically related but weakly coupled capabilities with limited overlap.
  • Correctness and adherence: Instruction adherence is not a reliable proxy for scientific correctness, and scientifically correct answers may still violate specified constraints.The findings support explicitly training and evaluating for joint attainment of both capabilities.

E.3 Results across Source Datasets

Results show substantial variation in instruction-following performance across source datasets and modalities, while added constraints generally reduce accuracy. Constraint-level analysis further reveals chemistry weaknesses and failures on fine-grained symbolic requirements.

  • Source-dataset variation: 63.92% DRFR on LLM4Mat-Bench versus 96.28% on MatSciBench shows substantial source-level variation within materials science.LLM4Mat-Bench is 22.55 percentage points below MatCha and 32.36 points below MatSciBench.
  • Modality analysis: Gemini-3.1-Pro-Preview is compared across multimodal and text-only subsets in Geography, Materials, and Physics.Chemistry and Biology are excluded because their samples are exclusively text-based.
  • Constraint effects: Constraint-augmented queries generally produce lower accuracy than original queries across most domains for GPT-5.2 and Qwen3.5-397B-A17B.The comparison associates added instruction requirements with an accuracy penalty.
  • Disciplinary weaknesses: GPT-5.2 performs significantly worse on chemistry constraints than on constraints from other disciplines.This aligns with chemistry being its lowest-scoring scientific field.
  • Fine-grained constraint failures: A physics response correctly satisfies analysis-step and response-structure constraints but violates the all-lowercase requirement.The example illustrates why constraint-level evaluation can isolate a specific failure despite correct reasoning and structural compliance.

F Data Compliance and Release

SciMIF’s release preserves source provenance and licensing conditions while providing processed data or reconstruction materials as permitted. The release also includes annotations, evaluation resources, and documentation intended to support reproducibility.

  • Data provenance: SciMIF augments samples from 13 existing scientific datasets while retaining the provenance of questions, visual inputs, and reference answers.Source datasets are used for academic research and processed under their respective licenses and terms of use.
  • Redistribution: For redistributable sources, processed samples are released directly with required attribution and license information.The release includes augmented queries, reference answers, constraint lists, task types, disciplines, and source identifiers.
  • Restricted resources: For restricted datasets or visual assets, SciMIF provides source identifiers, processing scripts, and reconstruction instructions instead of republishing the content.This approach allows eligible users to obtain original data from official sources.
  • Reproducibility resources: The public release includes the constraint taxonomy, annotations, evaluation configurations, verification scripts, judge prompts, model code, and source-license documentation.The stated release strategy supports reproducibility while respecting ownership, attribution, and redistribution conditions.
Loading 2608.25973v1…