Source-linked AI summary
BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering
Shun Ye, Vinny Chandran Suja, Chenlong Li, Chongming Jiang, Reza Zamani, Xiang Li, Christopher Bain, Yuqi Zhou, Walker Peterson, Huidong Wang, Chenglang Hu, Jongchan Park, Xiao Cheng, Benjamin Swedlund, Sandra Murillo, Anjali Sivanandan, Shiyu Sun, Liang Lanfeng, Mohammad Tariqul Islam, Baju C. Joy, Ishaq N. Khan, Sreedhar S. Kumar, Gabriel Mercado-Vásquez, James V. Vizzard, Jonathan M. Matthews, Helen Huang, Xiaolu Guo, Ethan Nicklow, Guorui Chen, Ryan A. Neff, Surjendu Maity, Hyeonjin Park, Han-ho Joo, Katherine Dong, Yuyan Cai, Weihang Huang, Yichen Zou, Rui Yan, Raphael Figueroa, Artem Goncharov, Bella Rose Schremmer, Lian Elsa Linton, Keisuke Goda, Liang Gao, Ke Cheng, Leonardo Morsut, Jennifer L. Wilson, Jianping Fu, Lim Chwee Teck, Deblina Sarkar, Andreas Hierlemann, Savaş Tay, Alexander Hoffmann, Donald Richieri Griffin, Jun Chen, Shana O. Kelley, Shyni Varghese, Jinwoo Cheon, Wilbur A. Lam, James J. Moon, Wilson W. Wong, Samir Mitragotri, Dino Di Carlo
TL;DR
Bioengineering lacked benchmarks assessing research-relevant experimental reasoning beyond factual recall. BioEVAL evaluates models across multiple task types, and results show uneven performance: experimental MCQs were harder for most models despite several cloud models exceeding 85% overall MCQ accuracy.
Problem
Bioengineering-specific benchmarks assessing experimental reasoning, literature comprehension, and multimodal interpretation in research-relevant contexts were lacking.
Source: §V · p. 33Method
BioEVAL combines expert-curated multiple-choice questions, literature-synthesis tasks, and pilot multimodal reasoning problems across major bioengineering subfields.
Source: III-5. Correctness-Pattern Agreement… · p. 29Results
Performance varied across task types: experimental MCQs were harder than conceptual questions for most models, while several cloud models exceeded 85% overall MCQ accuracy.
Source: III-5. Correctness-Pattern Agreement… · p. 29Takeaways & Limitations
BioEVAL provides a diagnostic map of current model strengths, unreliable areas, and priorities for bioengineering-oriented model development.
Source: III-5. Correctness-Pattern Agreement… · p. 29Literature-synthesis similarity does not establish factual accuracy and may miss quantitative magnitude or direction, while treating assertions absent from references as neutral.
Source: III-5. Correctness-Pattern Agreement… · p. 30
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing LLM benchmarking emphasizes factual recall, offering limited insight into model performance on frontier and multimodal tasks. We assembled BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institutional initiative designed to assess experimental reasoning capability across bioengineering (BE) subfields. BioEVAL spans 11 major BE subfields plus a set of uncategorized items, bringing together 22 research groups to create a PhD-level benchmark comprising 608 evaluation items: 1) 380 multiple-choice questions (MCQs, 359 retained after audit), 2) 218 literature synthesis tasks, and 3) 10 multimodal problems with experimental image interpretation. Benchmark items underwent authoring-group expert review and centralized quality control before evaluation. Following evaluation, a blinded cross-group consensus audit of the highest- and lowest-accuracy MCQ items flagged 21 questions for revision or removal; these were withheld, and all reported MCQ results are computed on the 359 retained items. We evaluated diverse cloud-scale foundation/multimodal models (e.g., ChatGPT, Gemini, and Grok) and locally deployable models suitable for inference on consumer-grade GPUs. Models achieved the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields. Leaderboard rankings characterize current capabilities, limitations, and development priorities across the evaluated BE task categories. BioEVAL is maintained as an extensible benchmark with standardized protocols for continuing expert item contribution and model evaluation.
III-3. Literature Synthesis Performance Across BE Subfields
BioEVAL’s literature-synthesis benchmark compared model-generated abstracts with author-written abstracts using semantic similarity, revealing substantial variation in performance across models and bioengineering subfields.
III-3. Literature Synthesis Performance Across BE Subfields
- The L-Syn benchmark comprised 218 expert-selected, abstract-withheld primary research articles; models generated publication-style abstracts that were compared with author-written abstracts using embedding-based semantic similarity metrics for precision and coverage.Source: III-3. Literature Synthesis… · pp. 16–17, Figure 4 · p. 17
- Similarity scores ranged from 0.09 to 0.72 across subfields; GPT-4o-mini and Gemini-2.5-Flash-Lite scored 0.63–0.72 across all subfields, while several other cloud models scored ~0.61–0.70.Higher scores indicate stronger agreement between generated and reference abstracts.Source: Figure 4 · p. 17, III-3. Literature Synthesis… · pp. 18–19
- Mismatched author-abstract pairs scored 0.222 ± 0.086 across different subfields, 0.301 ± 0.109 within a subfield across research questions, and 0.430 ± 0.114 within a research question; these are background references, not matched nulls.Source: Figure S8 · p. 19, III-3. Literature Synthesis… · p. 19
- Immunoengineering had the highest overall similarity, with top models reaching 0.72, while Bioimaging, Biophotonics, and Optics generally clustered at ~0.52–0.63; performance also varied within subfields.Because models differed in architecture, training data, scale, and inference strategy, the analysis did not identify which characteristics explained these differences.Source: III-3. Literature Synthesis… · p. 20
- Most cloud-scale systems scored above 0.60, whereas edge-deployable GPT-oss-20B scored below 0.6 across subfields, including 0.56 in Bioimaging, Biophotonics, and Optics; deepseek_vl2_tiny reached 0.09 in Drug Delivery, Therapy, and Nanomedicine.The study did not isolate effects of model scale, architecture, training data, or inference strategy.Source: III-3. Literature Synthesis… · pp. 20–21
III-4. Multimodal Reasoning Performance on Experimental BE Data
On BioEVAL’s 10-question multimodal pilot, four models each achieved 80% accuracy, but the results are descriptive rather than definitive. The section also distinguishes answer accuracy from explanation-text similarity and reports preliminary evidence that strong performance is not limited to cloud-scale models.
Multimodal Reasoning Performance
- The pilot comprises 10 expert-curated questions requiring image or graph interpretation plus BE knowledge, spanning experimental readouts from qPCR curves and gels to cell-culture images and enzyme-kinetics plots.The analysis is an initial assessment rather than a definitive model ranking.Source: III-4. Multimodal Reasoning… · p. 21
- Similar slopes and plateaus support roughly 64 times more initial template in red than black; IMAC enriched SWP5 near 44.5 kDa after 100 mM imidazole, and the culture was not ideal for inoculation.Cells appeared rounded or partially detached, with uneven density and dark clumps of debris, suggesting stress or possible contamination.Source: III-4. Multimodal Reasoning… · pp. 22–23
- Cloud-scale models generally had more stable explanation-text similarity scores, while Qwen-3.5-9B showed a broader distribution despite high answer accuracy.These scores measure semantic agreement of post-hoc explanations with expert explanations, not latent reasoning processes.Source: III-4. Multimodal Reasoning… · p. 24, Figure 5 · p. 25
- Several frontier cloud-scale models occupied the high-accuracy region, but Qwen-3.5-9B and other locally deployable models show that strong pilot performance is not restricted to cloud access.The authors describe these findings as preliminary and call for broader benchmarking before assessing downstream applications.Source: III-4. Multimodal Reasoning… · p. 24, Figure 5 · p. 25
- 80% accuracy: Gemini-2.5-Flash, Gemini-2.5-Pro, Kimi-K2.5, and Qwen-3.5-9B each answered 8 of 10 pilot questions correctly; rankings are descriptive, not precise estimates.Accuracy was highly sensitive to individual items because the pilot contains only 10 questions.Source: III-4. Multimodal Reasoning… · pp. 23–24, Figure 5 · p. 25
III-5. Correctness-Pattern Agreement and Ensemble Performance across LLMs
Exhaustively selected majority-vote ensembles achieved only modest in-sample gains over the best single model on BioEVAL’s MCQs. The strongest reasoning models shared correctness patterns, limiting the benefit of combining their votes.
III-5. Correctness-Pattern Agreement and Ensemble Performance across LLMs
- 0.916, 0.925, and 0.922 were the in-sample best 3-, 5-, and 7-model ensemble accuracies, versus 0.900 for the best single model.Naive top-3 matched the best single model at 0.900, while naive top-5 and top-7 each reached 0.911. The best five-model ensemble’s gain was approximately 9 additional correct responses among 359 MCQs; ensemble selection and evaluation used the same questions, so these are in-sample results, not estimates of generalization.Source: Figure 6 · p. 27, III-5. Correctness-Pattern Agreement… · p. 28
- On 332 MCQs with cross-model variation, κ clustering grouped 21 models into three clusters plus unclustered models; leading reasoning models tended to succeed and fail on many of the same questions.The 27 excluded invariant items comprised 26 universally correct and 1 universally missed question. Gemini-2.5-Pro, Grok-4-Fast-Reasoning, Grok-4-Fast, Kimi-K2.5, and DeepSeek-Reasoner formed the top-reasoner cluster.Source: III-5. Correctness-Pattern Agreement… · pp. 26, 28, Figure 6 · p. 27
- Fast or non-reasoning models and smaller generalists formed separate clusters, while low-accuracy unclustered models had near-zero κ with competent models but limited ensemble usefulness.The fast/non-reasoning cluster included Gemini-2.5-Flash, GPT-5-Nano, GPT-oss-20B, Qwen-3.5-9B, and LongCat variants; a mid-tier cluster included DeepSeek-Chat, Gemini-2.5-Flash-Lite, Ministral-14B-2512, Phi-4-Reasoning-Vision-15B, and GPT-4o-mini.Source: III-5. Correctness-Pattern Agreement… · p. 28
V. Method V-1. BioEVAL Study Design
BioEVAL is a global, multi-institutional benchmark designed to assess bioengineering LLMs beyond factual recall, including experimental reasoning, literature comprehension, and multimodal interpretation.
V-1. BioEVAL Study Design
- BioEVAL aims to establish an expert-curated benchmark and standardized evaluation procedures, identify model strengths and subfield-specific failure modes, and support future BE research workflows.The benchmark was motivated by the lack of BE-specific assessments of experimental reasoning, literature comprehension, and multimodal interpretation.Source: §V · p. 33
- BioEVAL evaluates domain knowledge and experimental reasoning with MCQs, scientific synthesis with literature tasks, and interpretation of experimental images, plots, or graphical results.MCQs include four answer options and expert-written rationales; literature tasks provide full-text articles with abstracts withheld and ask models to generate publication-style abstracts.Source: §V · pp. 33–34
- The community-driven benchmark spans 11 major bioengineering subfields plus contributed questions beyond those categories, with expert review and centralized quality control before evaluation.Review aimed to reduce ambiguity, remove poorly supported questions, and ensure a clearly defined correct answer or reference output.Source: §V · p. 34
- The study evaluated frontier cloud-scale, reasoning-focused, visual/multimodal, and locally deployable models using task-specific performance metrics and downstream ranking, comparison, and error analyses.MCQs used accuracy and explanation-text similarity; literature synthesis used semantic precision and coverage; multimodal questions used accuracy and explanation-text similarity. Models were available in Feb. 2026.Source: §V · p. 34
V-2. Selection of BE Subfields and Participating Expert Groups
BioEVAL selected representative bioengineering subfields by surveying research activity at leading global institutions, then combining measured prevalence with the feasibility of cross-group expert validation. The process yielded 11 subfields and an uncategorized question set for the first release.
V-2. Selection of BE Subfields and Participating Expert Groups
- The survey used global university rankings and manual departmental review to identify institutions for estimating contemporary bioengineering subfield representation.The authors assumed major global research institutions were among the most active contributors to academic bioengineering research; the initial shortlist contained approximately 120 institutions.Source: V-2. Selection of BE Subfields and… · p. 34
- Semantic grouping consolidated overlapping institutional research labels into 22 categories: 21 major bioengineering subfields and one category for less-represented areas.Institutional subfield labels were recorded in a structured spreadsheet and standardized using a Python-based semantic grouping workflow.Source: V-2. Selection of BE Subfields and… · pp. 34–35
- Recruitment prioritized higher-frequency subfields, and only fields with at least two participating groups were included to support independent expert review and consensus validation.This process selected 11 bioengineering subfields plus uncategorized questions; participating university groups worldwide contributed benchmark items in their areas of expertise.Source: V-2. Selection of BE Subfields and… · p. 35