VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

• Human pathology benchmarks have enabled systematic evaluation across diagnosis, grading, molecular prediction, and segmentation.Source: §1 · p. 1
• Preclinical toxicologic pathologists examine animal tissues across organs and dose groups to characterize drug toxicity and determine whether compounds can advance to human testing.Source: §1 · p. 1, §2 · p. 3
• Veterinary pathology lacks public benchmarks, while existing datasets emphasize isolated tasks and provide limited vision-language reasoning evaluation.Source: §1 · p. 2, §2 · p. 3
• VIPER contains 1,251 questions linked to 419 H&E-stained rat histology images across seven organ systems and three question formats.Source: §1 · p. 2, Figure 1 · p. 3
• Questions were authored and validated by ECVP board-certified veterinary pathologists, then refined through human-in-the-loop generation and adversarial filtering against mirage predictions.Source: §1 · p. 2
• The study also introduces TOXSCRIBE models and benchmarks veterinary-specialized, human pathology-specialized, and general-purpose multimodal systems.Source: §1 · p. 2

Benchmark composition

• VIPER comprises 1,251 question-answer pairs associated with 419 rat H&E regions of interest across seven organ systems and 23 named anatomic structures.Source: §3.1 · p. 4
• Board-certified veterinary pathologists validated annotations, while questions span MCQ, KPrim, and free-text formats to test multiple forms of multimodal reasoning.Source: §3.1 · p. 4, §3.2 · p. 4

Curation pipeline

• The curation workflow uses randomly extracted ROIs, morphology-based seed questions, LLM-generated variants, expert refinement, and adversarial filtering.Source: §3.2 · p. 4

Evaluation protocol

• MCQs use exact-match accuracy averaged across cyclic answer-order permutations, KPrim uses the ETH half-point rule, and free text uses a calibrated judge weighting accuracy 70% and completeness 30%.Source: §3.3 · p. 5

Validation and models

• A reader study sampled 100 image-question pairs and compared three ECVP-certified veterinary pathologists, while 16 models were evaluated under the same prompting and generation protocol.Source: §3.4 · p. 5, §4 · p. 5
• The study developed two veterinary-specialized models, TOXSCRIBEQwen and TOXSCRIBEGemma, alongside human pathology-specialized and general-purpose multimodal models.Source: §4 · pp. 5–6

Overall performance

• 62.4% overall score for TOXSCRIBEQwen and 61.2% for TOXSCRIBEGemma lead VIPER, significantly exceeding every non-veterinary-specialized model.The two TOXSCRIBE variants are statistically similar (P = 0.13), suggesting their domain-adaptation benefit transfers across backbones.Source: §5.1 · p. 6

Performance by question category

• 80.5% over-reading-probe performance for TOXSCRIBEGemma and 77.2% for TOXSCRIBEQwen exceed PathChat+ at 62.1% and GPT-5.4 at 55.1%.The category tests whether models resist hallucinating lesions in normal tissue; pathology identification remains challenging, with TOXSCRIBEQwen at 50.3%.Source: §5.1 · pp. 6–7

Performance by organ system

• TOXSCRIBE ranks first in six of seven organ systems, with gains of 9.5% in urinary and 6.0% in respiratory tissues.The narrowest reported margin is 2.0% in digestive tissue, and TOXSCRIBE outperforms PathChat+ in every organ system.Source: §5.1 · p. 7

Veterinary vs. human pathology

• TOXSCRIBEQwen exceeds PathChat+ by 11.5% overall, while veterinary reader VP2 scores 93.1% versus 72.4% for human-trained physician reader PP1 on MCQ.KPrim and free-text reader gaps are not significant, with differences of 7.6% and −0.8%, respectively.Source: §5.2 · pp. 7–8

The importance of visual grounding

• Removing images reduces scores by 41.7% for TOXSCRIBEQwen, 28.0% for PathChat+, and 29.0% for GPT-5.4 across VIPER.Reader MCQ performance also drops by 31.6% for VP3 and 35.7% for PP1 when images are removed.Source: §5.2 · p. 8

Gain from domain-specific fine-tuning

• 10.0% improvement over base Qwen3.5-27B and 6.8% over base Gemma 4 result from veterinary pathology-specific fine-tuning.Both gains are statistically significant at P < 0.001, and the training data were independent of VIPER.Source: §5.2 · p. 8

Challenges in understanding instructions

• Patho-R1-3B and Patho-R1-7B answer True on 90.2% and 87.6% of KPrim statements, while LLaVA-Med selects option A on 67.5% of MCQs.The biases occur despite competitive free-text scores for the Patho-R1 models and uniformly rotated correct-answer positions for LLaVA-Med.Source: §5.2 · p. 9

Oracle performance

• An oracle selecting the highest per-question score among TOXSCRIBEQwen, PathChat+, and GPT-5.4 reaches 78.2% overall.The models fail on substantially different questions, indicating complementarity across their predictions.Source: §5.2 · p. 9

Benchmark contribution

• VIPER provides an expert-curated benchmark for ROI-level vision-language evaluation in rat toxicologic pathology.Its design uses expert-authored questions, human-in-the-loop refinement, and adversarial filtering to emphasize visual grounding.Source: §6 · p. 9
• The benchmark extends pathology evaluation beyond human tissue by focusing on non-human toxicologic pathology.Source: §6 · p. 9

Empirical findings

• Human pathology-specialized and general-purpose frontier models show limited transfer to veterinary pathology.Source: §6 · pp. 9–10
• Veterinary-specific instruction tuning improves performance across different model backbones.Source: §6 · pp. 9–10

Reliability implications

• Normal-tissue performance provides a complementary reliability test because safety assessment requires detecting lesions while avoiding false positives in normal tissue.Source: §6 · p. 10

Scope limitations

• VIPER evaluates selected regions of interest rather than complete slide-level toxicologic pathology workflows.Slide-level review and dose-group comparison remain outside the benchmark’s current scope.Source: §2 · p. 15

Interpretation boundary

• VIPER performance should not be interpreted as evidence that models are ready for deployment in toxicologic pathology.Source: §2 · p. 15

Experimental details

• The paper reports benchmark composition, curation, evaluation, prompting, scoring, and statistical-analysis details needed to understand the main results.Source: §6 · p. 16

Statistical analysis

• Key comparisons use uncertainty and statistical-significance reporting, including agreement metrics and paired bootstrap testing.Source: §7 · p. 16

Compute requirements

• TOXSCRIBEQwen was trained on 8× NVIDIA H200 GPUs for approximately 90 hours, while TOXSCRIBEGemma required approximately 30 hours on the same hardware.Source: §8 · p. 16

Responsible use

• The paper discusses positive impacts such as improved evaluation and safer AI development, alongside risks from overinterpreting benchmark performance and inappropriate deployment.Source: §10 · p. 16
• VIPER is framed as a research evaluation resource rather than a clinical or regulatory decision system.Source: §11 · p. 17

Existing assets

• The paper credits original sources for preclinical data, benchmarks, baseline models, and software assets.Source: §12 · p. 17
• Images are redistributed under their source-repository licenses, including CC BY-SA 2.1 JP for Open TG-GATEs and CC BY-NC 4.0 for MMO.Source: §12 · p. 17

New assets

• VIPER documentation covers image sources, organ composition, question types, annotation, validation, intended use, licensing, and limitations.Source: §13 · p. 17

Human involvement

• The reader study involved three board-certified veterinary pathologists and one board-certified physician pathologist as research collaborators, without crowd workers or monetary compensation.Source: §14 · p. 17
• The paper states that human-subject IRB approval was not applicable because the study used preclinical animal pathology images and expert-authored or expert-validated questions.Source: §14 · p. 17

Model development

• TOXSCRIBEQwen and TOXSCRIBEGemma are LoRA fine-tuned versions of Qwen/Qwen3.5-27B and google/gemma-4-31B-it, using the same instruction-tuning corpus.Source: §A.1 · p. 19

Optimization

• The training pipeline uses AdamW, cosine scheduling, LoRA adapters, a frozen vision backbone, multi-magnification images, and 62,201 optimization steps.LoRA rank is 64, with α=128 and dropout 0.05.Source: §A.1 · p. 19

Benchmark composition

• VIPER covers seven organ systems and 23 named anatomic structures, with question counts organized by organ system and format.The benchmark includes MCQ, KPrim, and free-text formats.Source: §A.2 · p. 19

Question categories

• The question categories include anatomy identification, hallucination resistance, spatial localization, pathology identification, feature description, artifact recognition, and numeric extent estimation.Examples include identifying renal cortex, rejecting a false myositis claim, locating tubular necrosis, and estimating gland atrophy.Source: §A.2 · p. 20

Reader-study analysis

• MCQ and KPrim agreement are analyzed at choice and statement levels, while free-text reliability uses ICC(2,1) alongside Pearson correlation.The reliability summaries distinguish discrete-category agreement from continuous-score agreement.Source: §A.3 · p. 21

Score calibration

• TOXSCRIBEQwen scores 58.3 under GPT-5.4, 59.1 under replicate GPT-5.4, and 56.0 under Claude Opus 4.7.The absolute score changes with the judge while the model ordering is broadly retained.Source: §A.4 · p. 22

Judge agreement

• Pearson r = 0.98 and Spearman ρ = 0.95 show close item-level agreement between GPT-5.4 and Claude Opus 4.7 judges.Their disagreement is concentrated in absolute score levels rather than response ranking.Source: §A.4 · p. 22
• A replicate GPT-5.4 judge shifts model means by only 0.8–1.1 %, indicating small within-judge stochasticity.The replicate correlation with the headline pipeline is Pearson r = 0.99.Source: §A.4 · p. 22

Model comparison

• TOXSCRIBEQwen’s free-text advantage over PathChat+ is 5.4–5.7 % and statistically significant under both GPT-5.4 and Claude Opus 4.7.Holm-adjusted p ≤0.008 across the two-judge family.Source: §A.4 · p. 22

Statistical comparison

• Only one physician reader participated, limiting the generalizability of the KPrim and free-text null results.Source: §A.8 · p. 24

Image ablation

• The image-ablation matrix distinguishes Normal, Black, Random, and No image conditions and reports the Normal −No-image gap.Source: §A.9 · p. 25
• Random-image ablations are reported only for MCQ because KPrim and free-text random-image conditions used an earlier protocol.Source: §A.9 · p. 25

Cross-benchmark generalization

• PathChat+ leads PathMMU by 2.1% over ToxScribe (Qwen), while ToxScribe (Gemma) is statistically similar to GPT-5.4 at ∆= −0.7 pp.The cross-benchmark evaluation uses n = 8,468 questions.Source: §A.10 · p. 25

Generation prompt

• VIPER’s question-generation prompt targets rat H&E histopathology from TG-GATEs or MMO studies for veterinary pathology students and residents.The prompt supplies an image, seed question-answer pair, and organ, source, and magnification metadata.Source: §A.11 · p. 26, §1 · p. 26

2. Generation, user prompt.

• The metadata fields identify the source, rat organ, and magnification for each generated question.Source: §2 · p. 27
• Each generation request supplies a histopathology image, an expert seed question-answer pair, and instructions to produce MCQ, KPrim, and free-text variants.Source: §2 · p. 27
• Every question must require image examination, and question text must not name or describe the visible finding that determines the answer.Source: §2 · p. 27
• MCQ options must be plausible for the rat organ, similarly long, and checked for text-only answer leakage.Source: §2 · p. 27

3. Structured output schema.

• The structured schema defines MCQ stems, answers, explanations, and four distractors, plus KPrim stems with four true-or-false statements.Source: §3 · p. 28
• The schema defines KPrim questions with exactly four statements and free-text questions with a stem, expected answer, synonyms, scoring rubric, and maximum points.Source: §3 · p. 28

4. Adversarial text-only checker, MCQ, system prompt.

• The text-only MCQ checker must select exactly one option and return an answer index, confidence level, and reasoning as JSON.Source: §4 · p. 28

5. Adversarial text-only checker, MCQ, user prompt.

• MCQs require selecting one correct answer from five options and returning JSON only.Source: §5 · p. 28

6. Adversarial text-only checker, KPrim, system prompt.

• KPrim prompts require true/false judgments for all four statements, even when the image cannot be seen.Source: §6 · p. 28
• KPrim outputs must contain four Boolean answers, a confidence label, and brief reasoning in JSON format.Source: §6 · p. 28, §7 · p. 29

7. Adversarial text-only checker, KPrim, user prompt.

• New questions must test the same seed topic and remain grounded in the seed answer rather than changing subjects.Source: §7 · pp. 29–30
• To reduce guessability, questions emphasize image-specific appearance, location, distribution, counts, extent, and relationships.Source: §7 · pp. 29–30
• Distractors and statements are balanced to avoid textbook-answer, stem-hint, detail-level, or generally true-or-false cues.MCQ options are constrained to similar lengths, and failed items are regenerated after text-only readers guess correctly.Source: §7 · p. 29

10. Regeneration after a pathologist rejection, system prompt.

• The regeneration system positions the model as an expert veterinary pathologist and exam-question author.Source: §10 · p. 30
• The questions concern H&E-stained rat histopathology sections from TG-GATEs or MMO studies.Source: §10 · p. 30
• The intended audience is veterinary pathology students and residents.Source: §10 · p. 30
• Each regeneration receives a histopathology image, an expert-authored seed question and answer, metadata, and a reviewer-rejected question with feedback.Source: §10 · p. 30, §11 · p. 30

11. Regeneration after a pathologist rejection, user prompt.

• The prompt also provides other questions for the image so the replacement tests a different aspect.Source: §11 · p. 30
• The regenerated item must address reviewer feedback, avoid repeating the rejected approach, and reference the rat-organ image.Source: §11 · p. 30