PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVCBench: Benchmarking Evidence-Grounded Multimodal Reasoning Towards AI Virtual Cells

Manyu Li, Xunkai Li, Yongfu Xiong, Yi Liu, Rong-Hua Li, Guoren Wang

arXiv:2609.37773v1cs.AIq-bio.QM

TL;DR

Current virtual-cell benchmarks primarily evaluate cellular simulation, leaving systematic evaluation of experimental-evidence interpretation and hypothesis proposal less developed. OmniVCBench introduces a figure-centric benchmark with paired open-response and MCQ assessments and an evidence-referenced judge. Across eleven shared models, the formats correlate strongly, while the reported pilots and adaptation results remain bounded in scope.

  • Problem

    Existing AIVC benchmarks primarily evaluate simulation, while systematic evaluation of interpreting experimental evidence and formulating biological hypotheses remains lacking.

    Source: §1 · p. 2

  • Method

    OmniVCBench provides 6,077 source-traceable figure-based questions across inference, explanation, and hypothesis proposal, paired with AIVC-Judge scoring and model-derived MCQ distractors.

    Source: §1 · p. 3, §3.1 · p. 4, §3.3 · p. 6

  • Results

    Across eleven shared models, MCQ accuracy and open-response scores correlate at Spearman ρ = 0.964; GPT-5.6-sol scores 2.76 on L3 open responses.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Paired open-response and MCQ assessments provide complementary views of model performance, including distinct evidence-handling behaviors in case studies.

    Source: §5 · p. 10

    The benchmark’s narrow, dated source distribution bounds comparisons to that source and assay distribution.

    Source: E LIMITATIONS AND OUTLOOK · p. 33

Abstract

from arXiv · show

Artificial Intelligence Virtual Cells (AIVCs) are envisioned as scientific agents that simulate cellular responses, explain underlying mechanisms, and support hypothesis-driven discovery. Existing AIVC benchmarks, however, operate primarily at the simulation layer, motivating complementary evaluation of how models interpret experimental evidence and formulate biological hypotheses. We introduce OmniVCBench, a figure-centric, source-traceable benchmark for the interpretation component of an AIVC. It contains 6,077 curated single- and multi-subfigure question--answer pairs derived from figures and experimental contexts in the scientific literature. Guided by Bloom's taxonomy, we instantiate interpretation-layer counterparts of the AIVC Predict--Explain--Discover agenda through three scientific reasoning tasks. We further introduce AIVC-Judge, a task-conditioned MLLM-as-a-judge framework with category-specific, reference-aware rubrics for evaluating open-ended responses. A complementary Model-Derived Hard-Negative Mining (MDHNM) strategy converts plausible errors observed during model inference into MCQ distractors for lower-cost evaluation. Within the evaluated heterogeneous model pool, MCQ accuracy correlates positively with AIVC-Judge scores, providing a complementary view of performance alongside open-response evaluation. Code and data demo are available at https://anonymous.4open.science/r/OmniVCBench.

1 INTRODUCTION

AIVC evaluation has largely focused on simulating cellular states, leaving interpretation of experimental evidence and hypothesis proposal less systematically assessed. OmniVCBench evaluates this interpretation component with source-traceable questions and paired response formats.

1 INTRODUCTION

  • The interpretation component maps figures and questions to evidence-grounded answers; an agent adds planning, tool use, and iterative refinement.Its tasks derive observed results (L1), explain biological processes (L2), and propose evidence-grounded hypotheses (L3).Source: §1 · p. 2
  • OMNIVCBENCH provides 6,077 source-traceable questions spanning inference, explanation, and hypothesis proposal on cellular-research figures.The questions combine single- and multi-subfigure items with experimental contexts and linked source records.Source: §1 · p. 3
  • The benchmark pairs open-response and MCQ evaluation on identical items, with reference-aware scoring, model-derived distractors, and unanimous three-annotator item review.The paper also evaluates the component in executable evidence-acquisition pilots.Source: §1 · p. 3

2 RELATED WORK

OMNIVCBENCH complements simulation-focused virtual-cell benchmarks by assessing how models interpret visual experimental evidence. It builds on scientific-figure and biological-reasoning benchmarks while pairing response formats on cellular evidence.

2.1 VIRTUAL-CELL EVALUATION

  • Perturbation benchmarks assess simulation, while OMNIVCBENCH evaluates natural-language interpretation of observed cellular figures.Other language-based biology benchmarks discussed here reason over textualized omics profiles rather than visual evidence.Source: §2.1 · p. 3

2.2 SCIENTIFIC MULTIMODAL REASONING

  • OMNIVCBENCH builds on scientific-figure and biology-focused reasoning work with cellular figures, consistent L1–L3 tasks, and paired open-response and MCQ protocols.The paired protocols use identical items and support diagnostics of visual grounding, scoring agreement, and model errors.Source: §2.2 · pp. 3–4

2.3 TASK ORGANIZATION AND RESPONSE EVALUATION

  • Applying open-response and MCQ formats to the same cellular evidence supports comparison of answer selection with generated reasoning.MDHNM pools low-scoring open responses and rewrites them as candidate options.Source: §2.3 · p. 4

3 METHOD

OMNIVCBENCH organizes figure-based interpretation into three levels and evaluates the same image–question items through open responses and MCQs. Its construction, distractor mining, training resource, and judge are designed to retain links to experimental evidence.

3.1 TASK FORMULATION

  • L1 derives results from displayed observations, L2 explains biological processes, and L3 proposes evidence-consistent hypotheses with testable consequences.L3 assesses evidence-grounded proposal, not novelty or experimental confirmation; the benchmark uses the same image–question pair for open response and six-option selection.Source: §3.1 · pp. 4–5

3.2.1 OPEN-ENDED QA GENERATION

  • Two generator models produce candidate question–answer pairs from images and source text; independent cell-biology PhD annotators retain only unanimously approved items, totaling 6,077 of 7,923 candidates (76.70%).The audit yielded 6,077 items from 2,625 source records and 1,080 papers; unanimous decisions covered 93.15% of candidates, with Fleiss’ κ = 0.8560 [0.8442, 0.8674].Source: §3.2.1 · pp. 5–6

3.2.2 MODEL-DERIVED HARD-NEGATIVE MINING FOR MCQ GENERATION

  • MDHNM mines low-scoring model responses to the same image and question, rewrites them as plausible distractors, and audits the resulting options.Five distractors are selected using scientific incorrectness, relevance, uniqueness, and answer-length checks; a unanimous three-annotator audit applies to resulting items.Source: §3.2.2 · p. 6

3.2.3 OMNIVCTRAIN

  • OMNIVCTRAIN supplies coarse image–question–answer examples for supervised fine-tuning and retrieval, with benchmark-source exclusions retained for both uses.The training corpus contains 548,450 examples, and benchmark items receive additional manual validation.Source: §3.2.3 · p. 6

3.3 AIVC-JUDGE

  • AIVC-Judge checks response claims against source evidence, then returns rubric-specific dimension scores, an overall score, and claim-level evidence traces.On a stratified 300-item sample, human overall ratings correlated with the production judge at Spearman ρ = 0.877 across 884 paired responses; annotators lacked the original figures.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

Across evaluated models, MCQ and open-response scores are strongly associated, while case studies reveal inconsistencies between selected answers and explanations. Training yields modest MCQ gains, and evidence-acquisition pilots show mixed results under a bounded one-step setup.

4.1 SETUP

  • The evaluation covers four proprietary and eleven open-weight MLLMs, with eleven configurations also assessed through open responses.The open-weight models span 2B–8B parameters; the MCQ track also includes an SFT variant.Source: §4.1 · p. 7, Table 2 · p. 8

4.2 BENCHMARKING MLLMS WITH OMNIVCBENCH

  • GPT-5.6-sol leads at 55.1% MCQ accuracy and 3.28/5 under AIVC-Judge, but its open-response score falls from 3.68 on L2 to 2.76 on L3.Open-weight base models span 15.9%–33.8% MCQ accuracy and score below 2.40 on open responses.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 ADAPTATION WITH OMNIVCTRAIN

  • Rank-32 SFT improves average MCQ accuracy by 1.97 points, including an L3 increase from 29.6% to 34.8%; multimodal RAG improves it by 1.60 points.Both gains are significant under source-paper-clustered Holm-corrected analysis; the lower-rank configuration changes accuracy by +0.46 points.Source: §4.3 · p. 8, Figure 4 · p. 9
  • Joint image–question retrieval scores 35.41%, versus 33.62% for image-only and 33.40% for text-only retrieval; the strongest adapted Qwen configuration reaches 35.79%.The strongest adapted configuration remains substantially below the leading proprietary model.Source: §4.3 · pp. 8–9

4.2 BENCHMARKING MLLMS WITH OMNIVCBENCH

  • Across eleven shared models, MCQ accuracy and AIVC-Judge scores correlate at Spearman ρ = 0.964, but within four proprietary models Pearson r = 0.208.The pooled relationship includes strong separation between proprietary and open-weight systems, and individual rank inversions occur within proprietary models.Source: §4.2 · p. 8, Figure 4 · p. 9

4.4 ERROR ANALYSIS ON OMNIVCBENCH

  • Case studies reveal mismatches between MCQ selections and open explanations, including reversed effect directions and incorrect links between interventions, panels, and biological entities.L3 proposals can be testable yet misread the motivating observation, or conflict with the model’s own prediction or pathway diagram.Source: §4.4 · p. 9

4.5 EVIDENCE ACQUISITION IN THE AIVC REFINEMENT LOOP

  • In three numerical pilots, acquiring a selected measurement yields nine corrections and zero regressions, while self-review improves no pilot and degrades two.Real-microscopy classification shows no gain; these are one-step replay loops without simulator retraining, with corrections confined to queried readouts.Source: §4.5 · pp. 9–10

5 CONCLUSION

OmniVCBench evaluates the interpretation component missing from simulation-focused AIVC benchmarks, and the conclusion positions MLLMs as interpreters connecting predictions with experimental evidence. Evaluated models only partially fill that role, especially in evidence-grounded open-response argumentation.

5 CONCLUSION

  • MCQ accuracy and AIVC-Judge scores show strong ranking agreement, while case studies reveal distinct evidence-handling behaviors across the tracks.Human validation supports the shared-evidence scoring protocol, and OMNIVCTRAIN adaptation yields modest gains.Source: §5 · p. 10
  • The strongest model scores below three of five on L3 open responses, indicating weaker evidence-grounded argumentation.Source: §5 · p. 10
  • Four one-step evidence-acquisition pilots on real biological outputs position the MLLM as an interpretation component linking simulation with experimental evidence.The conclusion says current models fill this role only partially.Source: §5 · p. 10
  • The benchmark pairs source-traceable questions spanning inference, explanation, and hypothesis proposal with AIVC-Judge and model-derived distractors.Source: §5 · p. 10

AI USE STATEMENT

The appendix documents the benchmark’s task definitions, construction and curation, and evaluated model access routes; the AI-use statement describes how generative AI supported data production and result interpretation.

AI USE STATEMENT

  • Generative AI produced candidate question–answer pairs, distractors, and auxiliary task-type annotations, and supported result interpretation; retained items were approved by human annotators.Source: AI USE STATEMENT · p. 10

A.1 MODEL VERSIONS AND ACCESS ROUTES

  • Table 4 identifies evaluated systems, versions, access routes, construction models, and supporting models.API systems use provider routes, while open-weight checkpoints run locally from HuggingFace.Source: §A.1 · p. 16, Table 4 · p. 16

A.2 INTERPRETATION-LAYER TASK DEFINITIONS

  • The three interpretation levels have operational definitions, and P-E-D labels are functionally aligned rather than equivalent to those levels.Source: §A.2 · p. 16, Table 5 · p. 16, §A.4 · p. 18
  • Two independent annotator models assigned over 96% of the 1,738 L3 items to “propose novel”; the audit concerns task type, not construction-time level labels.The labelers’ observed agreement was 0.9937 and Cohen’s κ was 0.9068.Source: §A.2 · p. 16, Table 6 · p. 17

A.3 CONSTRUCTION FUNNEL

  • 6,077 of 7,923 candidates were retained after unanimous approval; annotators fully agreed on 93.15% of candidates, with Fleiss’ κ = 0.8560 [0.8442, 0.8674].The three cell biology PhD annotators reviewed candidates independently and were blind to one another’s judgments and generator identity.Source: §A.3 · p. 17, Table 7 · p. 17

A.4 SOURCE FILTERING AND QA CURATION

  • Candidate QA pairs undergo schema, source-evidence, answer-specificity, and visual-grounding checks before the unanimous three-annotator audit.L3 references anchor hypothesis assessment to source observations, while hypothesis validity depends on evidence consistency and testable consequences.Source: §A.4 · pp. 17–18, §A.3 · p. 17

A.5 DATASET COMPOSITION AND DIFFICULTY DISTRIBUTION

The benchmark spans multiple interpretation roles and literature-derived figure contexts, with composition and audit results clarifying its scope and the model-dependent contribution of images.

A.5 DATASET COMPOSITION AND DIFFICULTY DISTRIBUTION

  • 83.0% of retained items carry Analyze or Evaluate labels; B4 includes 1,542 inference and 1,763 mechanistic-explanation items, while L3 requests evidence-grounded hypotheses.The construction labels are not a validated measure of cognitive difficulty.Source: §A.5 · p. 18
  • The 6,077 benchmark items come from 2,625 records across 1,080 papers; 50.9% use multi-panel figures, and items reference a mean of 1.78 panels.The source coverage is 0.17% of the upstream corpus.Source: §A.5 · p. 18, Table 11 · p. 18
  • All benchmark papers are from Nature Communications and span 2011–2017; source papers are dominated by mouse and human studies, with microscopy and immunoblot the most frequent assays.Source: §A.5 · pp. 19–20

A.6 ITEM AND OPTION AUDITS

  • The response-derived difficulty statistic is descriptive within the evaluated model pool, not a measure of human answerability.It is available for 5,606 of 6,077 items; the remaining 471 were answered incorrectly by every model, leaving zero variance.Source: §A.6 · p. 20

A.7 MDHNM SCREENING CRITERIA

  • Every distractor originates from an observed model error; retained errors are minimally rewritten and screened before option ordering.MDHNM uses responses scored at most 2 by AIVC-Judge, with the original image included in scoring.Source: §A.6 · p. 20, §A.7 · p. 21

A.6 ITEM AND OPTION AUDITS

  • 22.6 percentage points: GPT-5.6-sol drops from 57.3% with images to 34.7% without; image contributions vary across models.InternVL3.5-2B and SmolVLM2-2.2B each score 13.7% without images, below the 16.7% random baseline.Source: Table 17 · p. 20, §A.6 · p. 20

A.7.1 DIRECT-GENERATION DISTRACTOR ABLATION

The appendix evaluates distractor difficulty, judge and human scoring, benchmark integrity, cross-track agreement, and adaptation resources. It reports a strong pooled MCQ–open-response relationship while qualifying its subgroup and rubric sensitivity.

A.7.1 DIRECT-GENERATION DISTRACTOR ABLATION

  • Directly generated distractors yield 30.3–38.0 percentage points higher GPT-5.6-sol accuracy than released MDHNM items, while matched Gemini-3.1-flash-lite comparisons are small and nonsignificant after Holm correction.The matched direct-minus-mined differences are +3.7pp for GPT-5.6-sol and +2.7pp for Grok-4.6; the reported construction comparison concerns the complete released configuration.Source: §A.7.1 · pp. 21–22, Table 19 · p. 21

A.8 AIVC-JUDGE RUBRICS AND PRODUCTION CONFIGURATION

  • AIVC-Judge uses level-specific scoring dimensions and caps decisive numerical, directional, entity, or evidence-contradiction errors; its outputs include claim evidence verdicts and dimension scores.The production configuration specifies the question, response, figure, caption, context, and reference answer as inputs, with candidate identity omitted.Source: §A.8 · p. 22, §A.14 · p. 28

A.9 HUMAN VALIDATION SAMPLE AND MCQ PERFORMANCE

  • On 300 validation questions, human MCQ accuracy ranged from 55.3% to 74.7%, with lowest level-wise accuracy on L3; option agreement was Fleiss’ κ = 0.589.The agreement estimate uses the 283 questions with valid answers from all three annotators.Source: §A.9 · p. 23, Table 24 · p. 23, Table 25 · p. 24

A.10 HUMAN SCORING OF OPEN-ENDED RESPONSES

  • The production judge correlated with the human panel mean at Spearman ρ = 0.877, and all three annotators and the judge ranked GPT-5.6-sol above Qwen3-VL-8B and LLaVA-Med-Mistral-7B.Agreement was lower for L3 (panel–judge ρ = 0.709), so the validation supports overall scores and ranking more strongly than fine-grained L3 dimensions.Source: §A.10 · pp. 24–25

A.11 PROPOSAL-ORIENTED RUBRIC SENSITIVITY PILOT

  • On identical full-source inputs, overall-score agreement was higher under the proposal rubric than the critique checklist (QWK 0.767 vs 0.466), and candidate rankings were preserved on the 17-item intersection valid for every annotator and condition.The evidence establishes neither a validated hypothesis-ability scale nor full-benchmark rescoring.Source: §A.11 · pp. 25–26

A.12 RELEASE-LEVEL TRAIN–BENCHMARK OVERLAP AUDIT

  • The released training and benchmark corpora had no confirmed overlap after source-, text-, and image-level checks, but the authors do not claim absolute leakfreeness.Perceptual image candidates were manually reviewed without confirming identical, cropped, or recomposed figures.Source: §A.12 · pp. 26–27

A.13 CLUSTER-ROBUST SCORES AND PAPER-WEIGHTED STATISTICS

  • Across eleven shared models, MCQ and open-response scores correlate strongly (Spearman ρ = 0.964), but the four proprietary systems show weaker correlation (Pearson r = 0.208).Leave-one-family-out Pearson correlations remain 0.922–0.971; removing single-model families is a leverage check, not a subgroup claim.Source: §A.13 · pp. 27–28, Table 35 · p. 28

B.1 SUPERVISED FINE-TUNING

  • The rank-32 SFT configuration improved Qwen3-VL-8B average MCQ accuracy by 1.97 points, while open-track scoring of the strongest MCQ configurations remains future work.The rank-32 subset configuration jointly differs in rank, data size, and batch size from the rank-16 full-corpus configuration; archived predictions reproduced the reported adaptation accuracies.Source: B OMNIVCTRAIN ADAPTATION DETAILS · p. 29, §B.1 · p. 29, Table 37 · p. 29

B.2 RETRIEVAL-AUGMENTED GENERATION

Joint image–question retrieval improves Qwen3-VL-8B MCQ accuracy relative to the base and unimodal retrieval controls. The appendix also describes pilots in which a model acquires biological evidence and revises predictions without retraining simulator weights or performing new wet-lab measurements.

B.2 RETRIEVAL-AUGMENTED GENERATION

  • Retrieval indexes jointly encode each image and question, exclude same-article and same-figure entries, and prepend three retrieved QA demonstrations as text while retaining the test image as input.Retrieved images are discarded, so this evaluates text-demonstration prompting conditioned on multimodal retrieval rather than interleaved multimodal in-context learning.Source: §B.2 · p. 29
  • Joint retrieval raises Qwen3-VL-8B accuracy from 33.82% to 35.41%; gains across three base models range from 0.76 to 1.60 points, with one change not statistically significant.At k = 3, image-only and text-only controls score 33.62% and 33.40%, respectively.Source: §B.2 · p. 30, Table 39 · p. 30, Table 38 · p. 30

C BIOLOGICAL EVIDENCE ACQUISITION AND PREDICTION REVISION

  • Four fixed-case pilots test evidence acquisition and prediction revision over biological outputs, archiving 108 calls across 36 cases and scoring initial, self-reviewed, and post-evidence answers.The model selects an additional measurement from reserved cells or images; simulator weights are not retrained and no new wet-lab measurement is performed.Source: C BIOLOGICAL EVIDENCE ACQUISITION AND… · p. 30, §C.1 · p. 30

C.2 RESULTS

The pilots test whether acquiring one chosen measurement can correct predictions in one-step replay, while showing that improvements remain local and do not establish a complete virtual-cell discovery loop.

C.2 RESULTS

  • Nine wrong-to-right and zero right-to-wrong transitions followed chosen measurements across three numerical pilots; self-review improved no pilot and degraded two.Initial accuracy was 66.7%–93.3% across the four pilots.Source: §C.2 · p. 30, Table 41 · p. 31, Figure 6 · p. 32
  • All four pilots began below perfect accuracy, but the microscopy pilot showed no classification gain when reading real fluorescence pixels.The initial accuracy range was 66.7%–93.3%.Source: §C.2 · p. 30
  • Corrections were local: unqueried-readout accuracy did not improve after calibration, and simulator weights remained frozen in one-step replay.The update changed a response-calibration layer rather than the simulator.Source: §C.2 · p. 30, §C.3 · p. 31

C.3 SCOPE OF THE PILOTS

  • Small case counts, no new wet-lab measurements, and no independent gold standard for mechanisms limit the pilots to replayed prediction revision.The archive contains 36 cases and 108 model calls; free-text mechanisms remain explanation traces.Source: §C.3 · pp. 31–32

D AIVC-JUDGE DIMENSION-WISE RESULTS

Proprietary models lead on every reported AIVC-Judge dimension in the evaluated pool. The dimension scores describe rubric-based behavior, with weaker human agreement on L3 argumentation and evidence weighing than on overall scores.

D AIVC-JUDGE DIMENSION-WISE RESULTS

E LIMITATIONS AND OUTLOOK

The limitations bound interpretation to a dated, literature-derived evidence distribution and to the benchmark’s specified scoring and system settings. The case studies offer qualitative diagnoses of errors and divergence between open responses and selected options.

E LIMITATIONS AND OUTLOOK

  • Inference is bounded by a 2011–2017 Nature Communications corpus and MLLM-only evaluation; adaptation results cover LoRA and retrieval augmentation on one 8B backbone.The source corpus is dominated by mouse and human studies, microscopy, and immunoblot assays; observed adaptation gains are configuration-dependent.Source: E LIMITATIONS AND OUTLOOK · pp. 32–34
  • L3 measures bounded hypothesis proposal, not experimental execution or discovery novelty, while 34.7% no-image MCQ accuracy indicates substantial text-derived signal.The reported with-image accuracy is 57.3%, a 22.6-point gap; text-derived signal includes question text, options, domain priors, and possible literature memorization.Source: E LIMITATIONS AND OUTLOOK · p. 34
  • Human scoring agreement checks shared textual evidence and rubric application, not the judge’s image reading; same-input expert scoring remains unvalidated.L3 agreement is lower, and its critique-oriented dimensions can conflate proposal quality with unrequested critique.Source: E LIMITATIONS AND OUTLOOK · p. 34

F CASE STUDIES OF REASONING AND SCORING

  • The nine purposively selected cases are qualitative diagnoses, showing relational reasoning errors and distinct diagnoses across open responses and MCQ selections.Cases A–G concern response-level evidence and reasoning errors; H and I examine flawed selections alongside defensible open hypotheses.Source: F CASE STUDIES OF REASONING AND SCORING · pp. 34–35

F.1 SINGLE- VERSUS MULTI-SUBFIGURE DIFFICULTY

  • All five representative models were at least as accurate on multi-subfigure as single-subfigure items, with differences of 0.4–7.8 percentage points.For GPT-5.6-sol, accuracy rose from 56.0% to 63.7% on L1 and 58.0% to 62.6% on L2, and changed from 44.7% to 44.1% on L3.Source: §F.1 · p. 35, Table 43 · p. 35

F.2–F.4 CASES A–C

  • Cases A–C expose polarity, counterfactual-premise, and cross-model direction errors; a correct choice can still coexist with a faulty explanation.In Case A, the measured outcome is the number of centrin-positive structures, not mature functional centrioles.Source: §F.2 · p. 36, §F.3 · p. 37, §F.4 · p. 38

F.5–F.7 CASES D–F

  • Cases D–F require preserving intervention-reversal, reciprocal-perturbation, and protein-pair spatial relations across panels.The Case D evidence supports an inhibitor-sensitive change in the assay but does not identify a unique direct molecular target.Source: §F.5 · p. 39, §F.6 · p. 40, §F.7 · p. 41

F.8 CASE G: HAIR-REGENERATION HYPOTHESIS ON A FAULTY PREMISE (L3)

  • Case G distinguishes contrasting regeneration outcomes from hypotheses about cellular roles, and flags an unsupported observational premise or necessity claim.Reported pigmentation recovery was 68.3% with SB and 40.1% with PHM; normal morphology was 27.0% and 83.3%, respectively.Source: §F.8 · p. 42

F.9 CASE H: YAP–ECT2/FGD3 PAIRED-TRACK DIVERGENCE (L3)

In Cases H and I, open answers propose testable extensions of displayed evidence, while selected options can contain direction or mechanism errors. The paired tracks therefore diagnose different aspects of the same item.

F.9 CASE H: YAP–ECT2/FGD3 PAIRED-TRACK DIVERGENCE (L3)

  • In Case H, open answers propose testable interventions from expression evidence, while GPT-5.6-sol’s selected option reverses the direction implied by its own hypothesis.Expression association does not establish direct promoter binding or a loss-of-function effect.Source: §F.9 · p. 43

F.9–F.10 CASES H–I

  • Both cases show that an evidence-consistent open hypothesis can accompany a flawed MCQ selection, so the two response formats carry different diagnoses.In Case H, option F also contradicts its own stated mechanism claim.Source: §F.9 · p. 43, §F.10 · p. 44

F.10 CASE I: INTEGRIN–LCK PAIRED-TRACK DIVERGENCE (L3)

  • In Case I, GPT-5.6-sol’s open answer proposes a controlled intervention test, but its selected option contradicts the integrin→Lck initiation shown in panel h.The displayed panels report Lck inhibition and knockdown, not the proposed β1-integrin antibody experiment.Source: §F.10 · p. 44

G ETHICS, LICENSE, AND DATA AVAILABILITY

OmniVCBench is derived from figures and experimental contexts in peer-reviewed publications, with no new biological experiments or identifiable information. Its license, research-only intended use, and staged release define how the materials may be used.

G ETHICS, LICENSE, AND DATA AVAILABILITY

  • All benchmark items come from figures and experimental contexts in peer-reviewed publications assembled through OmniScience; no new biological experiments were conducted.The benchmark contains no personal or otherwise identifiable information.Source: G ETHICS, LICENSE, AND DATA AVAILABILITY · p. 45
  • OMNIVCBENCH and OMNIVCTRAIN use the CC BY-NC-SA 4.0 license, while redistribution of source figures remains subject to original publishers’ terms.The released packages host materialized figure crops, and the authors impose no additional or differing terms of use.Source: G ETHICS, LICENSE, AND DATA AVAILABILITY · p. 45
  • The benchmark is intended only for research evaluation; its annotations, scores, and model outputs are not clinical or experimental decision-making evidence.Evaluated models were accessed through public APIs or open-weight releases under their respective licenses.Source: G ETHICS, LICENSE, AND DATA AVAILABILITY · p. 45
  • The repository provides the AIVC-Judge implementation and a 300-item demo pairing open-response QA items with MDHNM MCQs; the complete datasets are slated for release after review.The full releases comprise 6,077 OMNIVCBENCH items and 548,450 OMNIVCTRAIN items.Source: G ETHICS, LICENSE, AND DATA AVAILABILITY · p. 45

Related papers

Loading 2609.37773v1…