Source-linked AI summary
PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning
Taegyun Kim, Youngwook Ham, Jungwook Rhim, Ju-Hyun An, Sungkyu Park, Kunwoo Park
TL;DR
Veterinary AI evaluation lacks realistic resources despite increasing demand for pet-health advice. The paper introduces PETQA, a Korean long-form benchmark built from expert-answered dog and cat questions, and evaluates eighteen models across zero-shot, RAG, and SFT settings. Closed models generally perform better, multimodal questions are consistently harder, and adaptation gains are inconsistent, motivating more reliable veterinary AI evaluation and methods.
Problem
Veterinary benchmarks remain limited despite the need to assess models’ knowledge and clinical reasoning for realistic pet-health questions.
Method
PETQA provides a Korean long-form benchmark of real-world dog and cat questions with expert-veterinarian answers, text-only and multimodal data, and annotated test sets, evaluated across zero-shot, RAG, and SFT settings.
Results
Closed models generally outperform open-weight models, multimodal questions consistently reduce performance, and RAG and SFT yield inconsistent improvements.
Takeaways & Limitations
The findings characterize current veterinary clinical QA capabilities and underscore the need for more effective methods for reliable veterinary-care support.
Takeaways & Limitations
PETQA focuses on dogs and cats, leaving broader species coverage and cross-language generalizability for future work.
Abstract
from arXiv · showhide
We introduce PetQA, a Korean long-form question-answering (QA) benchmark for evaluating veterinary knowledge and clinical reasoning in large language models (LLMs) and large vision-language models (LVLMs). PetQA contains 10,076 text-only and 8,751 multimodal QA pairs derived from real-world questions about dogs and cats, paired with answers from expert veterinarians. Its test split, PetQA-Bench, further includes annotations for question types and clinical conditions. We evaluate eighteen models using ROUGE, BERTScore, and LLM-as-a-judge metrics for factuality and helpfulness under three settings: zero-shot inference, retrieval-augmented generation (RAG), and supervised fine-tuning (SFT). The benchmarking results provide an overview of the strengths and limitations of current models in addressing veterinary clinical queries and highlight the need for more effective adaptation methods to develop clinically reliable AI systems for veterinary care. To facilitate broader use, we additionally provide translated versions of PetQA-Bench in five languages.
1 Introduction
Veterinary clinical QA lacks benchmark resources despite growing use of AI for pet health questions. PETQA addresses this gap with a Korean long-form dataset and initial evaluations of model capabilities and adaptation strategies.
- Veterinary AI benchmarks have largely focused on human medicine, leaving limited resources for evaluating veterinary knowledge and clinical reasoning.
- PETQA contains 10,076 text-only and 8,751 multimodal QA pairs about dogs and cats, with answers from verified veterinary experts.
- PETQA-BENCH includes 2,000 test QA pairs per modality with question-type and clinical-condition annotations for systematic evaluation.
- The benchmark evaluates eighteen models with ROUGE, BERTScore, factuality, and helpfulness under zero-shot, RAG, and SFT settings.
- Closed models generally outperform open-weight models, multimodal questions remain harder, and RAG and SFT produce inconsistent improvements.
- PETQA is publicly released with translated PETQA-BENCH versions in five languages to support broader research.
2 Related Work
Medical QA benchmarks commonly use article-based, multiple-choice, or short-answer formats, while veterinary resources remain scarce. PETQA extends evaluation toward long-form veterinary QA for realistic dog and cat clinical queries.
- Existing medical benchmarks include article-based span or binary questions and multiple-choice questions derived from licensing examinations.
- Multiple-choice benchmarks enable straightforward scoring but may overestimate clinical reasoning through superficial answer-choice patterns.
- Long-form QA has been explored to require detailed answers to open-ended medical questions.
- Veterinary benchmark resources remain scarce and have been limited to multiple-choice question answering.
- PETQA provides a Korean long-form veterinary benchmark for comprehensive evaluation of systems answering dog and cat clinical queries.
3 Dataset: PETQA
PETQA is constructed from real-world Korean pet-care questions about dogs and cats, paired with expert answers and organized into text-only and multimodal benchmark sets. Its annotations and descriptive analyses capture question types, clinical conditions, and dataset diversity.
- Dataset construction: PETQA combines text-only and multimodal questions about dogs and cats in realistic clinical scenarios with answers provided by expert veterinarians.
- Data collection: The source platform contributed veterinary questions, answers, and images from a large collection of Korean community QA posts.
- Data preprocessing: Rule-based preprocessing retained verified-expert answers for dog and cat posts after removing duplicates and corrupted images.
- Data preprocessing: An LLM-based pipeline filtered irrelevant or speculative content, removed personally identifiable information, corrected text, and removed promotional material.
- Quality control: Manual evaluation found 97 of 100 sampled instances coherent and 92 preserving all essential information after LLM-based preprocessing.
- Label annotation: PETQA-BENCH labels questions by type and, for diagnosis questions, annotates clinical conditions using an expert-reviewed annotation process.
- Quality control: Expert review found 99 of 100 sampled QA pairs relevant and 85 correct, supporting the dataset’s overall reliability.
- Descriptive analysis: Diagnosis is the most frequent question type, and the benchmark covers 352 unique clinical conditions across its Text and Multimodal sets.
4 Experiments
The experiments benchmark eighteen Korean-supporting models across zero-shot, RAG, and SFT settings using lexical, semantic, factuality, and helpfulness metrics.
- The study compares zero-shot inference, RAG with a veterinary clinical reference, and SFT using the PETQA training set.
- Eighteen models were evaluated, grouped into closed LVLMs, open-weight LVLMs, and open-weight LLMs.
- Four metrics measure lexical overlap, semantic similarity, factual alignment, and reference-free helpfulness.
- ROUGE-L measures lexical overlap, while BERTScore captures semantic similarity between generated and reference answers.
- Factuality uses Gemini 2.5 Flash to compare atomic facts with references, whereas GPT-4o rates helpfulness on a five-point Likert scale.
5 Benchmarking Results
Benchmarking shows that closed models generally lead, multimodal questions are harder, and adaptation methods produce metric- and modality-dependent trade-offs.
- Zero-Shot Performance: Closed models achieved the highest performance in most cases across the Text and Multimodal sets.GPT-4.1 mini led Text ROUGE at 0.303 and BERTScore at 0.746, while Gemini 2.5 Pro led Multimodal ROUGE at 0.246 and helpfulness at 4.814.
- Zero-Shot Performance: Factuality and helpfulness generally increased with model size, unlike ROUGE and BERTScore, which showed no consistent size relationship.
- Zero-Shot Performance: Multimodal performance was consistently lower than Text performance even after matching question-type distributions.The remaining difference may reflect greater multimodal difficulty, current LVLM limitations, or both.
- Performance by Question Type: Question type significantly affected every metric–set combination, with Text ROUGE and BERTScore strongest for basic knowledge and factuality strongest for diagnosis.Closed LVLMs consistently outperformed other groups in helpfulness across question types.
- Clinical Condition Inclusion: Gemini 2.5 Pro ranked first for clinical-condition inclusion, and matched diagnosis responses exceeded unmatched responses on all four primary metrics.For Gemini 2.5 Pro, matched versus unmatched responses scored 0.296 versus 0.277 in ROUGE, 0.744 versus 0.739 in BERTScore, 0.607 versus 0.571 in factuality, and 4.947 versus 4.929 in helpfulness.
- Retrieval-Augmented Generation: RAG improved factuality for 12 of 18 Text models but reduced ROUGE for 12 and BERTScore for 11, demonstrating inconsistent adaptation effects.On the Multimodal set, factuality and helpfulness each improved for 8 of 11 models, while ROUGE and BERTScore decreased for 8 and 9 models.
- Supervised Fine-Tuning: SFT improved Text ROUGE and BERTScore for all 14 models but decreased factuality for 11 and helpfulness for all 14.On the Multimodal set, ROUGE improved for all 7 models and BERTScore for 6, while factuality decreased for 6 and helpfulness for all 7.
- Supervised Fine-Tuning: After SFT, Qwen3-VL-32B responses became shorter, more generic, and more conservative, improving surface similarity while reducing explanatory richness and coverage.Across settings, factuality improved only for Gemma-2-27B and MedGemma-27B on Text, and MedGemma-27B on Multimodal.
6 Conclusion
PETQA introduces a multilingual, expert-answered benchmark for veterinary clinical QA and evaluates current models across text and image-based cases. The results show model and adaptation limitations relevant to clinically reliable veterinary AI.
- PETQA combines text-only and image-based dog and cat questions with answers from expert veterinarians based on real-world cases.
- PETQA-BENCH adds question-type and clinical-condition annotations and is presented as the first long-form veterinary medicine QA resource.
- Translated PETQA-BENCH versions are released in English, German, Chinese, Indonesian, and Arabic.
- Closed models generally outperformed open-weight models, multimodal questions consistently reduced performance, and RAG and SFT yielded inconsistent improvements.
- The benchmark highlights current model limitations and the need for more effective adaptation methods for clinically reliable veterinary AI systems.
Limitations
PETQA’s coverage is limited to dogs and cats, and its reference-based evaluation depends on the accuracy of expert-provided answers. The authors address answer reliability through veterinary manual validation but identify broader species coverage and expert quality checks as future needs.
- Dataset coverage: The dataset focuses on dogs and cats, limiting its species coverage and leaving generalizability to broader species for future work.The authors note that dogs and cats represent a substantial share of real-world pet-related queries and that future work could extend coverage.
- Evaluation metrics: Reference-based evaluation may be misleading when expert-provided reference answers are incorrect.The study therefore verified the medical accuracy of collected responses through manual veterinary review.
- Quality assurance: Manual validation by a veterinary expert with a Ph.D. was used to assess the medical accuracy of online answers.The authors suggest incorporating expert-based quality checks into future resource-construction efforts.
Ethical Considerations
The study documents institutional approval, privacy protections, licensing constraints, data-use restrictions, and model-term compliance. Its artifacts and source materials are framed for non-commercial research and evaluation.
- Research approval: The study received Institutional Review Board approval from Soongsil University.The approval identifier is SSU-202604-HR-805-1.
- Privacy and licensing: PETQA is released for research purposes under CC BY-NC-ND 4.0, which restricts commercial use and derivative-work distribution.The authors manually confirmed that PETQA-BENCH contains no personally identifiable information.
- Annotators: PETQA-BENCH annotation was conducted by six AI students from two institutions and was classified as minimal risk.The annotators comprised four master’s students and two undergraduates and were compensated according to local wage regulations.
- Data sources: The dataset uses publicly accessible Naver Knowledge iN veterinary QA data after review of the platform’s terms of service and posting policy.The source materials include user-generated posts and expert answers collected under the platform’s stated conditions.
- External resources: The retrieval corpus was an English veterinary reference book used only as an external knowledge source for retrieval-augmented evaluation, not model training.The book was intended for professional veterinary education and clinical practice, but its title could not be disclosed because of legal constraints.
- Reference and model compliance: Clinical-condition annotation used Korean veterinary and medical references internally for annotation and terminology normalization, without redistributing their contents.The study also documents compliance with the applicable terms for the models used in evaluation.
B Dataset Details
PETQA data were collected from Naver Knowledge iN and processed through structured collection, animal-type classification, and dataset-statistics procedures. The resulting benchmark distinguishes dog- and cat-related content for downstream evaluation.
- Raw data collection: Raw QA posts were collected from Naver Knowledge iN using keyword- and verified-expert-profile strategies.Posts were crawled with Selenium and parsed with BeautifulSoup before extracting question, answer, selection, and board information.
- Animal type classification: Animal type was assigned through a two-stage pipeline combining hashtag-based weak labels with text classification.Posts matching only dog- or cat-specific hashtag lists were labeled directly; remaining posts were classified as dog, cat, or neutral using fine-tuned KLUE-RoBERTa-base models.
- Dataset statistics: Training and validation set statistics are reported in Table A1.The provided passage identifies the table as the source of descriptive statistics but does not state the individual values.
C Evaluation Metrics
The evaluation combines reference-based overlap, semantic similarity, factuality, and reference-free helpfulness measures, with reproducible judge protocols. It also uses partial matching to evaluate clinical-condition spans.
- Evaluation protocol: LLM-based evaluations used random seed 42 and temperature 0 to support reproducibility and consistency.Evaluations were conducted through the batch API, with prompts provided in Appendix G.
- Reference-based metrics: ROUGE uses morpheme-level Korean tokenization with Okt and reports mean F1 over examples.Reference and generated answers were tokenized with KoNLPy before ROUGE computation using the rouge Python package.
- Reference-based metrics: BERTScore uses the multilingual bert-base-multilingual-cased checkpoint and reports mean F1 over examples.The metric was computed through the official bert-score Python API.
- LLM-as-a-judge metrics: Factuality compares atomic facts in reference and generated answers, reporting mean F1 to capture accuracy and completeness.Gemini 2.5 Flash served as the judge; precision measures generated facts supported by the reference, while recall measures reference facts covered by the output.
- LLM-as-a-judge metrics: Helpfulness is evaluated reference-free by having a judge assign a score and brief rationale using detailed criteria and examples.The judge receives the question and model response rather than a reference answer.
- Judge-model selection: GPT-4o was selected as the helpfulness judge after human-score comparison, matching human scores exactly for 58 samples and differing by at most one point for 38 more.The authors caution that helpfulness results require further validation because the observed correlation was modest.
- Clinical-condition evaluation: Partial matching uses longest common substring overlap between predicted and gold clinical-condition spans to accommodate minor span variations.EXAONE-3.5-32B extracts predicted conditions before partial precision and recall are computed.
D Experimental Setups
The experiments evaluate models under controlled inference, retrieval, and fine-tuning settings, with supplementary analyses comparing representative responses across settings.
- Computing environment: Experiments used two computing environments with either three RTX A6000 GPUs or one H200 GPU.The systems had 128 GB and 2 TB of system memory, respectively.
- RAG: RAG retrieved the top six passages from a FAISS index using Qwen3-Embedding-8B embeddings and 1,024-token chunks with 100-token overlap.
- SFT: Parameter-efficient fine-tuning used LoRA with rank 16, dropout 0.05, α=32, and two training epochs.Training used AdamW with a learning rate of 5 × 10^-5 and batch size 8.
- Response analyses: Supplementary analyses compared representative successful and failed responses between zero-shot and RAG settings, and between zero-shot and SFT settings.
F Supplementary Results
Supplementary analyses examine clinical-condition inclusion, factuality components, enhanced reasoning, multilingual evaluation, and controlled comparisons between text and multimodal questions. They also report translation quality and validation procedures that qualify interpretation of the results.
- Clinical condition inclusion: Gemini 2.5 Pro performed best on multimodal clinical-condition inclusion, followed by GPT-4.1 and Gemma-3-27B.The analysis used partial match scores for clinical conditions in model responses.
- Matched-group comparison: 0.259>0.241 ROUGE, 0.731>0.723 BERTScore, 0.574>0.484 factuality, and 4.836>4.764 helpfulness favored the matched group.
- Question-type control: Text remained higher-performing than Multimodal after restricting analyses to diagnosis questions and matching question-type distributions, with all paired comparisons significant at p<0.05.The results suggest the performance gap is more likely related to greater multimodal-question difficulty than to question-type distributions.
- Factuality breakdown: Precision was lower than recall in 26 of 29 factuality comparisons, indicating unsupported information despite coverage of much reference content.
- Enhanced reasoning: Enhanced reasoning consistently improved helpfulness, while ROUGE, BERTScore, and factuality varied across models and modalities.
- Multilingual benchmarking: Translation validity rates were 98% for English and Indonesian, 97% for German and Chinese, and 96% for Arabic.The rates came from 100 randomly sampled instances per target language.
- Multilingual benchmarking: Closed models generally achieved the highest multilingual performance, but cross-language score differences were not interpreted because metrics were not implemented identically.LVLMs underperformed on the Multimodal set in most cases.
- Annotation procedure: Question-type annotations used automatic exact matching followed by manual additions for surface-form variations in clinical-condition names.