PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniScientist: An Omni-Modal Omni-Discipline AI Scientist

Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, Wynne Hsu

arXiv:2608.13558v1cs.AIcs.CL

TL;DR

Existing AI-scientist systems often reduce scientific evidence to text or scalar summaries, losing spatial, temporal, and procedural relations. OmniScientist instead uses direct multimodal perception with autonomous agents across ideation, experimentation, and writing, completing all 36 evaluated cases with a mean paper score of 6.3.

  • Problem

    AI-scientist systems that reduce evidence to text or scalar summaries leave scientifically relevant spatial, temporal, and procedural relations unavailable.

  • Method

    OmniScientist combines a perception layer with autonomous agents for ideation, experimentation, and writeup, while code-enforced checks support novelty, validity, provenance, and claim traceability.

  • Results

    All 36 cases produced compiled papers, with a mean overall score of 6.3 across a seven-dimensional rubric.

  • Takeaways & Limitations

    Direct multimodal perception supports evidence-grounded scientific discovery across disciplines, evidence families, and modalities.

Abstract

from arXiv · show

Recent advances in foundation models have enabled AI scientists to automate increasingly complete research workflows, from hypothesis generation and code execution to manuscript preparation. Yet workflow coverage alone does not provide access to the full evidence on which scientific discovery depends. Existing systems typically reason over text, code, labels, or precomputed summaries, leaving scientifically decisive spatial, temporal, cross-channel, and procedural relations unavailable to the agent. We introduce OmniScientist, an end-to-end, omni-modal AI scientist that conducts multidisciplinary research directly from heterogeneous raw evidence. A perception layer and 3 autonomous agents for ideation, experiment, and writeup operate within a deterministic pipeline, allowing observations to shape research questions, experimental decisions, and final claims throughout the research lifecycle. By running idea, rigour, and claim checks in code, the system enforces novelty screening, statistical validity, execution provenance, and numerical traceability. We evaluate OmniScientist on 36 real-data cases spanning 5 discipline families, 4 families of scientific evidence, and modalities including images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs. The system completes the full path from raw data to a compiled manuscript in all 36 cases and achieves a mean overall paper score of 6.3 with the reference reasoning backbone. In paired comparisons against a blind variant that receives only precomputed scalar features, direct perception improves all 7 evaluation dimensions and wins 85% of head-to-head judgments. These results show that lifecycle-wide perception is essential for evidence-grounded scientific discovery and provides a practical path toward broadly capable AI scientists.

1 Introduction

OmniScientist addresses the gap between workflow automation and access to the heterogeneous spatial, temporal, statistical, and procedural relations in scientific evidence. It combines direct perception with autonomous agents for ideation, experimentation, and writing, and completes end-to-end research across a broad 36-case suite.

  • 1 Introduction: Existing multimodal benchmarks usually fix observations and questions in advance, while scientific agents typically use perception only at local workflow stages.This limits access to evidence that could shape research questions, experiments, and scientific writing across the full workflow.
  • 1 Introduction: The system combines a perception layer with 3 autonomous agents for ideation, experiment, and writeup, using observations to guide the research process.Within each stage, a ReAct loop interleaves observation, reasoning, and action.
  • 1 Introduction: OmniScientist completed the full path from raw data to a compiled paper in all 36 cases spanning 5 discipline families, 4 evidence families, and diverse scientific modalities.The suite included images, waveforms, audio, video, 3-D structures, trajectories, tables, formulae, and graphs.
  • 1 Introduction: OmniScientist works directly from heterogeneous scientific evidence so spatial, temporal, statistical, and procedural relations remain available throughout discovery.These relations are carried by modalities such as microscopy, spectra, waveforms, audio, video, 3-D structures, distributions, and trajectories.

2 Related work

Prior work combines autonomous scientific agents, general multimodal perception, and agentic reasoning, but prompt-mediated orchestration leaves stage transitions and validation dependent on fallible model output. OmniScientist instead uses code-controlled transitions to ground observations, verify stages, and backtrack when needed.

  • Autonomous scientific discovery: Scientific agents progressed from single-domain, single-instrument laboratory procedures to cross-disciplinary systems proposing hypotheses for subsequent bench validation.Early examples include diffraction-guided synthesis and laboratory automation, while later systems used real scientific data across disciplines (Mitchener et al., 2025; Villaescusa-Navarro et al., 2025; Swanson et al., 2025; Gottweis et al., 202
  • Multimodal scientific perception: General multimodal models provide strong image-text and vision-language perception, with scientific benchmarks evaluating charts, figures, microscopy, and materials observations.The cited foundation methods use contrastive image-text pretraining (Radford et al., 2021) and instruction-following vision-language models (Alayrac et al., 2022; Liu et al., 2023).
  • Agentic reasoning: Agentic reasoning supports autonomous discovery through tool use, verbal reflection, multimodal observations, and specialised roles that divide scientific work among agents.The passage identifies single-agent reasoning and tool use (Wei et al., 2022; Yao et al., 2023; Schick et al., 2023), reflection (Li et al., 2026a; Shinn et al., 2023), multimodal action (Yang et al., 2023b), and shared knowledge and review (Shao et al., 2025; Li et al., 2026b).
  • OmniScientist’s position: OmniScientist replaces prompt- and message-dependent orchestration with a code-controlled agent pipeline that verifies grounding in real observations and backtracks when a stage is unsupported.Reasoning within each stage remains open-ended, while code controls transitions and validation around it.

3 Problem setting

OmniScientist is framed as an evidence-grounded research task over heterogeneous raw artifacts, addressing the limitations of systems that mainly process text and numerical data. Its domain-agnostic pipeline accepts a single specification and applies the same perception-to-writeup loop across 5 discipline categories and 36 second-level cases.

  • Motivation: Current AI-scientist systems mainly process text and numerical data, leaving perceptual and procedural evidence unexamined and narrowing their questions and disciplinary reach.Token serialization does not guarantee that local spatial, temporal, cross-channel, or procedural relations survive; textual captions can lose such structure.
  • Evidence scope: OmniScientist is designed to perceive 4 families of scientific evidence spanning artifacts such as images, symbolic structures, numerical results, and experimental-process records.
  • Task formulation: The task supplies one specification file describing a dataset, scientific subject, target property, and corresponding raw data, while leaving methodology to the agent.The required output is an evidence-grounded paper, with nothing else supplied.
  • Evaluation scope: The demonstration suite covers 5 top-level discipline categories and 36 second-level cases, enabling evaluation of the engine’s domain agnosticism.
  • Domain generality: Adding a discipline requires only an additional specification file because the unchanged engine reuses the same perception, ideation, experimentation, and write-up loop without domain-specific code.The loop is described as operating across seismograms, CAD meshes, and knowledge graphs.

4 The OmniScientist framework

OmniScientist is an end-to-end AI scientist that combines raw-evidence perception with autonomous ideation, experimentation, and writeup agents. Its deterministic pipeline uses code-enforced checks to ground hypotheses, experiments, results, and manuscript claims in executable evidence.

  • Perception layer: The perception layer organizes raw artifacts by evidence family and modality, prioritizing native numeric analysis and invoking visual rendering only when spatial or structural patterns matter.Visual inspection is budget-constrained to balance thorough analysis with computational efficiency.
  • Framework architecture: The framework combines a perception layer with three autonomous agents for ideation, experiment, and writeup, processing four evidence families across twelve modalities.Raw evidence enters a sequential pipeline in which ideation formulates hypotheses, experimentation tests them, and writeup compiles the manuscript.
  • Ideation: Ideation uses a ReAct loop to inventory materials, inspect observations, search literature, and formulate computationally answerable, novel, falsifiable questions before code-enforced validation.Validation requires a research question, hypothesis, experiment sketch, falsification criterion, five self-filtered candidates, and at least three focused literature searches.
  • Experiment: Experimentation iteratively generates and debugs code, uses perception to inspect inputs and plots, and requires at least four analyses including controls such as baselines, ablations, mechanism probes, or sensitivity sweeps.An exit check verifies dataset access, execution provenance, figure correspondence, and multiple-comparison correction across all attempted tests.
  • Writeup: The writeup stage adapts manuscript structure and length to venue and discipline, retrieves references through OpenAlex, audits claims against the experimental record, and compiles the final PDF.Its five structural specifications support discipline-specific conventions, including sections for machine learning, biomedical, and chemistry papers.

5 Evaluation

OmniScientist completes end-to-end manuscript generation across a 36-case suite, achieving high and broadly consistent quality across backbones, disciplines, and evidence modalities. Direct multimodal perception improves grounding and significance, while prior-art search and iterative agentic reasoning are key contributors to overall quality.

  • 5.1 End-to-end evaluation: 6.3 overall score: Claude-powered OmniScientist returns complete manuscripts across the full 36-case suite, with strong alternate backbones performing in a similar range on evaluated subsets.The comparison swaps only the reasoning backbone while keeping the three roles separate, and Table 4 reports dispatches, completions, and mean composite scores across the suite.
  • 5.1 End-to-end evaluation: 6.1–7.1 median composite scores: generation quality remains stable across discipline families and evidence modalities, with top-scoring manuscripts spanning all domain categories.The reported range covers groupings by both discipline and evidence modality, supporting broad generalization across the evaluated cases.
  • 5.3 Perception ablation: +2.8 multimodal grounding and +1.8 significance: direct perception produces the largest gains over text-only scalar-feature baselines, while factual accuracy remains equally high.The grounding improvement is attributed to processing raw observations directly, which text-only baselines cannot access.
  • 5.4 Component ablations: 6.9 to 5.7: removing prior-art search causes the steepest leave-one-out performance decline, while reducing the iterative agentic loop to one pass also significantly degrades manuscript quality.The prior-art omission increases the risk of redundant or previously published ideas, identifying search and iterative reasoning as critical components.
  • 5.5 Qualitative analysis: 70%–87%: excluding ties, the perception-driven system wins across all 7 evaluation metrics, with near-zero ties for grounding, significance, and novelty but roughly one-quarter for accuracy and reproducibility.The paired qualitative analysis found that perception-driven research questions relied on attributes exclusive to raw multimodal records.
  • 5.6 Robustness and scaling: 2.9-point gaps: factual accuracy and soundness are most sensitive to backbone scale, whereas multimodal grounding shifts only 1.2 points across the same largest-to-smallest model range.The results indicate that stronger backbones improve some evaluation dimensions more than others, while scaling alone does not provide perceptual competence.

6 Conclusion

OmniScientist is an end-to-end, omni-modal, discipline-agnostic AI scientist that integrates multimodal perception throughout the research lifecycle, enabling raw observations to guide research and support manuscript claims.

  • OmniScientist integrates multimodal perception directly into the research lifecycle, linking raw observations to ideation, experimental execution, and manuscript claims.
  • The framework demonstrates cross-disciplinary applicability by successfully completing a 36-case demonstration suite spanning 5 discipline families.
  • The system is presented as an end-to-end, omni-modal, and discipline-agnostic AI scientist.

A Additional evaluation detail

This section provides supporting evaluation analyses covering computational cost, backbone performance, perception gains, and judge validation. It also specifies the fixed review rubric used consistently across cases and backbones.

  • Supporting analyses: The supplementary evaluation covers per-paper cost by backbone, backbone-score heatmaps, dimension-level perception gains, and judge-panel validation metrics.These analyses support the primary evaluation in Section 5.2.
  • Computational cost: The experiment stage dominates overall computational cost across all evaluated systems.Table 11 reports cost per paper by backbone; open-weight backbones were run locally.
  • Perception gains: Table 12 reports perception-enabled minus blind-baseline panel-score differences for each evaluation dimension across five blind pairs and one vision-off pair.Scores use the original 0–10 scale; positive values indicate improved performance with visual perception, and Cardiology uses the weaker vision-off run.
  • Judge validation: Judges score seven dimensions from 1–10 using a fixed rubric that emphasizes novelty, soundness, clarity, significance, reproducibility, multimodal grounding, and factual accuracy.Each judge receives the same rubric, manuscript source, figure captions, and authors’ result ledger for every case and backbone, with validation metrics and preset acceptance thresholds reported separately.
  • Backbone performance: The backbone heatmap shows reasoning-strength ordering, with stronger models at the top and smaller open-weight models at the bottom, while factual accuracy is consistently highest.Darker cells indicate higher panel scores, and factual accuracy forms the darkest column across evaluated models.

B The checks enforced in code

OmniScientist enforces idea, rigour, and claim checks as deterministic Python predicates over each stage’s finalize payload and accumulated loop state. Rejections become new observations that force continuation until acceptance or the step budget is exhausted, exposing result-selection as the dominant failure mode.

  • B The checks enforced in code: Each check is a Python predicate receiving the stage’s finalize payload and accumulated loop state, returning acceptance or a reason string rather than model self-assessment.The claim check evaluates the drafted manuscript itself, while the other predicates operate on stage-finalization records.
  • B The checks enforced in code: Rejected finalizations append the failure reason as a new observation, so the agent continues reasoning and the stage cannot end until acceptance or the step budget runs out.The gated-stage algorithm applies the predicate over what actually happened in the run and admits the payload only after a successful check.
  • B The checks enforced in code: Two thirds of all rejections concerned result selection rather than arithmetic, while outright fabrication was rare with one provenance rejection across 36 runs.Most remaining rejections involved schema discipline, including empty verdicts, unset lead selection, missing key numbers, and under-counted tests.
  • B The checks enforced in code: 115 finalize attempts were rejected across 36 primary runs, and only 4 runs reached both stage exits without being sent back.Result selection dominated: 26 runs tried to present a non-significant analysis as a finding and were forced to demote it.

C Run-level execution statistics

Across 36 primary runs, ideation was search- and perception-heavy while experiment was execution-heavy, and the rigour check preserved negative or weak findings rather than promoting them.

  • Run composition: Ideation spent most calls on literature searches and raw-item inspection without running code, whereas experiment averaged 31.8 run_python calls per run and only occasionally revisited raw evidence.The question-shaping observation had usually already been made by experiment time.
  • Run composition: No run exhausted the 24-step ideation or 50-step experiment budget.These budgets and the non-exhaustion result are reported for the 36 primary runs.
  • Run outcomes: 67 analyses were executed, found wanting, and deliberately demoted from the manuscript, representing roughly a fifth of everything the system computed.Demoted analyses remained in the execution trace while being excluded from the manuscript by the claim check.

D The perception layer

The perception layer exposes modality-native numeric and visual readers, letting agents inspect heterogeneous evidence without forcing every modality into images. Tools are automatically unlocked from each case’s specification file and were used across the 36 primary runs.

  • Perception architecture: Modality-native readers provide numbers in each modality’s own terms alongside visual readers, allowing the agent to choose between numeric analysis and artifact inspection.Signals, audio, video, 3-D structures, and trajectories each receive both native and visual reading capabilities.
  • Perception architecture: Tools are automatically unlocked from the specification file, requiring no per-discipline registration across the perception suite.Table 18 reports tool availability and invocation counts across the 36 primary runs.
  • Perception tools: The suite includes signal, audio, 3-D, trace, trajectory, and video analyzers exposing modality-specific measurements such as FFT frequencies, RMS, PCA axes, path length, and frame-difference motion.The listed tools were invoked 40, 39, 21, 20, 19, and 3 times respectively in the reported runs.

E Task specification for a new discipline

OmniScientist adds a new discipline through a single specification file, without modifying the engine. The specification defines the scientific role, data subject, measured property, open request, and dataset members while leaving methods and hypotheses to the agents.

  • E Task specification for a new discipline: A new discipline requires one specification file and no engine changes, with the engine reading nothing else about that discipline.The file contains roughly 1,500 dataset members in the seismology example, while its members list includes only file paths and shipped metadata.
  • E Task specification for a new discipline: The specification’s four scientific fields are the agent’s role, data subject, measured property, and open research request.The file names no method, hypothesis, or analysis, leaving those choices to the research workflow.
  • E Task specification for a new discipline: In the seismology case, the agent analyzes 60-second three-component broadband waveforms sampled at 100 Hz from the STEAD catalogue.Each record measures ground-motion amplitude over time on orthogonal E, N, and Z components, with earthquake/noise labels and accompanying station and source metadata.
  • E Task specification for a new discipline: The seismology request asks the system to find a novel, testable question, choose a method, run real waveform code, and produce a short publishable paper.The member records provide signal files, labels, station metadata, and earthquake magnitude, distance, and depth when applicable.

F Field-specific writeup specifications

The writeup stage resolves five field-specific structural specifications from the case specification or inferred subject. Each section receives a word budget, paragraph count, and paragraph-level outline, which the system expands using only the allocated experiment-record slice.

  • F Field-specific writeup specifications: Five structural specifications define the writeup stage’s field-specific section skeletons and abstract word ranges.Table 19 presents the five specifications and gives the word range for a single-paragraph abstract.
  • F Field-specific writeup specifications: Chemistry writeups use Introduction, Experimental Section, Results and Discussion, and Conclusions as their structural sections.The listed chemistry structure has a 150–250 word range in the supplied specification.
  • F Field-specific writeup specifications: The system resolves specifications from the case specification or inferred subject, then assigns each section a word budget, paragraph count, and paragraph-level outline.The drafting pass expands the outline one section at a time using only the experiment-record slice allocated to that section.

G Stage prompts

Ideation and experiment use runtime-assembled, discipline-agnostic prompts whose demands are enforced as predicates at stage exit. The prompts ground ideas in inspected data and constrained claims, while requiring comprehensive, honest, and statistically rigorous experiments.

  • Prompt design: Runtime-assembled prompts use case specifications and detected modalities, while remaining independent of any named discipline, modality, or method.Ideation and experiment share a template structure across disciplines; the writeup stage has no comparable single text.
  • Prompt design: Every prompt demand is also an exit-check predicate, so advancing a stage does not depend solely on the model choosing to comply.The exit check refuses the stage when a required demand is missing.
  • Ideation stage: Ideation requires inspecting representative materials, conducting focused literature searches, deriving questions from observed data, and grounding novelty and claims conservatively.The grounding rules require reconciling visual interpretations with given labels, avoiding unsupported novelty claims, separating direct evidence from broader mechanisms, estimating statistical power from real data, and preventing label leakage.
  • Experiment stage: The experiment prompt specifies a full-study battery covering primary, baseline, ablation, mechanism, breakdown, sensitivity, and grouping analyses when supported by the data.Grouping requires group-disjoint or leave-one-group-out evaluation alongside random-split metrics when grouping identifiers are present.
  • Experiment stage: The experiment stage forbids fabricated results and enforces multiplicity correction, independence checks, honest power language, and anti-HARKing selection of the strongest supported lead.If the data cannot support a test, the prompt requires reporting it as infeasible; weak or failed analyses are not promoted as leads.
Loading 2608.13558v1…