Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation

• WSIs’ gigapixel scale, sparse and weakly localized findings, abstract reports, and variable reporting styles create a spatial–semantic gap and make reliable image–text alignment difficult [68] [67] [82].Source: §1 · p. 3
• Existing WSI report-generation resources are limited: TCGA reports need extensive parsing, HISTAI emphasizes pretraining, and HANCOCK focuses on one cancer type, hindering standardized evaluation and cross-organ generalization [50].Source: §1 · p. 3
• The CAP-guideline-curated dataset contains approximately 10,500 WSI–report pairs from five institutions and is presented as the first large-scale Pan-Asia-centric dataset of its kind.Source: §1 · p. 4
• Pretrained VLM backbones with effective multimodal fusion consistently achieved superior report-generation quality, while conventional approaches struggled to capture report structure.Source: §1 · p. 4
• Pan-Asia-trained models generalized strongly to independent European cohorts, while clinical precision, linguistic coherence, and completeness of pathological reasoning correlated strongly with overall performance.Source: §1 · p. 4
• REG 2025 is established as the first computational pathology benchmark jointly evaluating report generation and vision-language understanding, while offering practical insights for developing future multimodal pathology models.Source: §1 · p. 4

2.1. Dataset Description and Preprocessing Pipeline

• Reports were standardized to a unified template, reviewed by trainees and board-certified pathologists, and limited to image-observable diagnoses and findings.Terminology followed the WHO Classification of Tumours and relevant CAP Cancer Protocols, with consensus harmonization for stomach low-grade dysplasia.Source: §2.1 · p. 5
• 10,494 WSI–report pairs across seven organs formed the final cohort, collected from five institutions and spanning malignant, premalignant, benign, and non-neoplastic diagnoses.Technical quality screening and expert review excluded diagnostically equivocal cases, leaving a cohort 15.4% smaller than the initial collection.Source: §2.1 · pp. 5–6

2.2. Challenge Design and Workflow

• Approximately 8,500 paired samples were provided for training, followed by two test phases with 1,000 WSI-only cases each and patient-level partitioning.Phase 1 used Pan-Asia cases; Phase 2 combined 500 Pan-Asia and 500 European cases to assess performance across ethnic and institutional domains.Source: §2.2 · p. 6

2.3. Evaluation Metrics

• The clinically grounded evaluation combined ROUGE-L and BLEU-4 lexical overlap with keyword Jaccard similarity and biomedical sentence-embedding cosine similarity.The composite addressed the limits of lexical overlap and keyword matching alone, including wording variation and clinically important negation.Source: §2.3 · pp. 6–7

3. Results

• The challenge attracted 389 participants from 40 countries, with 20 teams submitting in Phase 1 and 22 in Phase 2.Phase 1 had 8 individual and 12 group teams; Phase 2 had 9 individual and 13 group teams.Source: §3 · p. 7

3.1. Quantitative Evaluation Results

• Phase 1’s top score was 0.8098 (ICGI; average 0.5915), while Phase 2’s was 0.8494 (IMAGINE Lab; average 0.6523); all three leading teams scored above 0.8000.The authors report that Phase 2 improved over Phase 1 and that performance on European cohorts suggests potential cross-regional generalization.Source: §3.1 · p. 7
• German-subset performance had higher means and lower variance than Pan-Asia, with rankings largely preserved; in Phase 2, BLEU and KEY sharply declined among teams whose Ranking Scores were below 0.7000.The authors attribute the cross-subset differences to dataset composition and variability, not population-level difficulty, and say Ranking Score balanced penalization and reward.Source: §3.1 · pp. 7–8

3.2.1. Diagnostic Concordance: Attribute Fidelity and Robustness

• Generated reports showed diagnostic concordance: ICGI captured DCIS type, nuclear grade, and necrosis status, and structured prostate diagnoses, though prostate tumor volume was overestimated.The qualitative assessment examined diagnostic content and contextual features of WSIs against ground-truth reports.Source: §3.2.1 · pp. 8–9, §3.2 · p. 8
• Models correctly generated several rare diagnoses, but gastric lymphoma errors reflected histomorphologic overlap, while cervical outputs captured colorectal-type morphology despite incorrect anatomic sites.Rare cases included lymphoma, breast metaplastic carcinoma, and fundic gland polyp; MTS_REG correctly identified gastric malignant melanoma where most models predicted chronic gastritis.Source: §3.2.1 · pp. 9–10

3.2.2. Diagnostic Discordance: Quantitative Miscalibration and Overcalling

• 80% vs. 5% in the ground-truth report: three high-scoring teams overestimated tumor volume despite correctly identifying acinar adenocarcinoma, Gleason score, and grade group.The reported REG scores were 0.9749–0.9751; missed carcinoma yielded 0.3866–0.4056, and the REG score tracked the hierarchical severity of diagnostic errors.Source: Table 4 · p. 9, §3.2.2 · p. 10
• Models often replaced uncertain or broader diagnoses with definite labels, including overcalling ADH as DCIS and specifying fibroepithelial lesions as fibroadenoma or phyllodes tumor.The authors also observed difficulty translating visual estimates into rule-based grading for breast-biopsy mitotic activity and tubule formation.Source: §3.2.2 · p. 10

3.3. Benchmark Comparison with Existing Approaches

• Public benchmark models performed approximately 16.9% worse in Phase 1 and 30.4% worse in Phase 2 than leading challenge submissions; top methods used structured representation, grounding, or intermediate reasoning.The paper argues that report generation requires maintaining hierarchical diagnostic relationships and clinically grounded concept consistency, beyond direct sequence generation from slide-level features.Source: §3.3 · pp. 10–11

4. Overview of Submitted Algorithms

• Of 13 teams scoring at least 0.6000 invited to submit descriptions, 11 responded; most used foundation models and shared report-processing, WSI-preprocessing, and architecture stages.Common report preprocessing included normalization and hierarchical organization; many teams reused Trიდენტ, CLAM, or Prov-Gigapath, while most architectures used ABMIL or Transformers.Source: §4 · p. 11

4.1. Preprocessing Strategies

• WSI preprocessing was largely consistent across participants and showed no significant performance differences attributable to preprocessing strategies alone.Most teams used standardized MIL-based frameworks such as Trident or CLAM; some also accounted for organ variation or selected salient tiles. By contrast, the top three teams—ICGI, ICL_PathReport, and IMAGINE Lab—structured report categories; ICGI aligned tagged sections to tile features, ICL_PathReport used a hierarchical annotation tree, and IMAGINE Lab generated category prompts.Source: §4.1 · pp. 11–12
• Category-aware methods improved BLEU by +0.09 and KEY by +0.13 from Phase 1 to Phase 2, versus +0.01 and +0.02 for tokenization; Phase 2 gaps were 0.14 BLEU and 0.18 KEY.The three top teams using category-specific structuring averaged ROUGE 0.8572 and BLEU 0.6853 in Phase 2; four teams treating reports as structured semantic maps averaged KEY 0.7990. Explicit report structuring was linked to better cross-domain generalization and more stable clinical-term preservation.Source: §4.1 · p. 12

4.2.1. ABMIL-Based Architectures

• ICGI and ADCT used ABMIL slide embeddings with text decoders, but ICGI’s n-gram constraints supported rare-case predictions while ADCT misclassified a breast lymphoma as metaplastic carcinoma.ICGI’s constrained Transformer decoder was intended to suppress hallucinations and sometimes captured subtle morphology; ADCT used a cancer-specific codebook and knowledge distillation, with organ anchoring preserved in the cited lymphoma error.Source: §4.2.1 · pp. 12–13

4.2.2. Tree-of-Experts-Based Architectures

• ICL_PathReport’s Tree-of-Experts separated organ, diagnosis, and subtype information, supporting fine-grained attributes but making outputs vulnerable to incorrect upstream organ routing.In a breast lymphoma case, the model generated “prostate lymphoma”; the reported behavior suggests it selected the wrong organ branch before choosing a morphologically compatible diagnosis.Source: §4.2.2 · p. 13

4.2.3. Cross-Modality Transformer-Based

• Cross-modality Transformers used cross-attention to align WSI image tokens with report text, but outputs could remain plausible despite incorrect organ assignment and varied in language quality.IMAGINE Lab added organ- and diagnosis-level prompts, IUCompPath used a Cross-modal Context Module, and TeamTiger@REG2025 preserved patch-level spatial information with a Position-Aware module.Source: §4.2.3 · p. 13

4.2.4. Vision-Language Model-Based Architectures

• VLM-based teams supplied WSI features to biomedical language models, including BioGPT or BioBART, for pathology report generation.PathoMozhi used Flamingo-style gated cross-attention for vision-language alignment, while NW-TIA augmented BioBART with multi-task auxiliary heads to improve training stability and preserve WSI-feature semantics.Source: §4.2.4 · pp. 13–14

4.2.5. Retrieval-Augmented Generation (RAG)-Based Architectures

• MTS_REG_2025’s retrieval-augmented approach improved accuracy on rare cases but showed reduced robustness on high-prevalence cases.It retrieved semantically similar reports for Qwen3-8B; adaptive retrieval adjusted document counts by similarity gap to avoid redundant or insufficient retrieval and support consistency and semantic alignment.Source: §4.2.5 · p. 14

Metric limitations

• KEY and EMB aligned more consistently with final rankings than ROUGE and BLEU, which sometimes diverged from clinical correctness despite high or low lexical scores.Some models missed critical diagnostic terms despite high ROUGE, while others preserved clinically important attributes despite low BLEU.Source: §5 · p. 14
• Because key attributes determine clinical interpretation, pathology-specific evaluation should assess concept correctness through ontology-aware, slot-level, or clinically grounded consistency measures.The KEY score partially addresses the limitation of conventional metrics, which do not sufficiently evaluate concept-level correctness.Source: §5 · p. 14

Generation limitations

• Models recurrently hallucinated numeric attributes and misclassified diagnoses, generating plausible but ungrounded values because language models predict tokens probabilistically.The problem affected tumor proportion, grading score, and subtype classification; multimodal alignment alone was insufficient, motivating explicit diagnostic reasoning.Source: §5 · p. 14

Reasoning-based generation

• Slot-based prediction [74] [14], structured chain-of-thought [72] [70] [83] [81], and VQA decomposition [31] support stepwise reasoning, with VQA approaches reducing hallucination and improving slot grounding.The discussion presents structured reasoning-based generation as more suitable for pathology reports than conventional end-to-end captioning.Source: §5 · p. 14

Knowledge integration

• Retrieval-augmented generation and hierarchical expert architectures reduced hallucination and improved semantic grounding by drawing on external evidence or structured expert knowledge.The discussion recommends external knowledge integration alongside reasoning-aware multimodal architectures for future pathology foundation models.Source: §5 · p. 15

Future directions

• The challenge frames pathology report generation as multimodal reasoning involving structured diagnosis, quantitative attribute estimation, and uncertainty-aware decisions, prioritizing grounded metrics, reasoning-integrated generation, and structured pathology knowledge.Source: §5 · p. 15

6. Conclusion

• REG 2025 constructed and publicly released the first large-scale Pan-Asia multi-institutional pathology dataset, comprising approximately 10,500 WSI–report pairs, and systematically compared participating teams’ models.Source: §6 · p. 15
• Structured report representations and multimodal grounding were associated with superior performance, while VLM frameworks generated clinically coherent reports and maintained stable performance on an independent European cohort.The challenge identifies multimodal alignment, structured report representation, and clinically meaningful feature integration as key factors influencing report-generation performance.Source: §6 · p. 15
• The REG Challenge treats pathology report generation as a multimodal diagnostic understanding problem rather than image captioning and provides a benchmark and foundation for future computational pathology research.Its comparative analysis highlights multimodal alignment, structured report representation, and clinically meaningful feature integration.Source: §6 · p. 15

A.1. ICGI (Fig. S2-(a))

• ICGI uses H-optimus-1 to extract tile-level embeddings for its pathology-reporting pipeline.The pipeline applies color correction and tissue detection before feature extraction.Source: §A.1 · p. 19
• A multiple-instance slide-level embedding supports prediction of anatomical site and procedure.The team uses the representation for both classification tasks.Source: §A.1 · p. 19
• The pipeline generates slide descriptions with a decoder and post-processes the text to refine its output.The decoder is a Transformer decoder fed by the slide-level representation.Source: §A.1 · pp. 19–20
• The ABMIL backbone uses eight parallel attention heads whose weighted sums are concatenated into a slide-level representation.This embedding is then passed to task-specific prediction heads.Source: §A.1 · p. 20
• Two linear classifiers predict site and procedure, while a Transformer decoder generates text from the slide-level embedding.These are the three task-specific heads in the pipeline.Source: §A.1 · p. 20
• At inference, an n-gram mask derived from the training corpus filters out unobserved token sequences.Source: §A.1 · p. 20

A.2. ICL_PathReport (Fig. S2-(b))

• The Tree-of-Experts framework hierarchically predicts report components, including organs, procedures, and histological types, using specialized lightweight MLP classifiers.The final report is synthesized by navigating the tree’s forward path, preserving the hierarchy of clinical pathology reports.Source: §A.2 · p. 20
• Noise perturbation and weighted Mixup augment slide features for rare or complex diagnoses, with Noisy Feature Mixup intended to improve generalization and prediction calibration.Source: §A.2 · p. 20
• The pipeline extracts patches with Trident, encodes them with Virchow, aggregates features using the PRISM Perceiver, and synthesizes reports through the Tree-of-Experts forward path.Source: §A.2 · p. 20

A.3. IMAGINE Lab (Fig. S2-(c))

• IMAGINE Lab combines CONCH, TITAN, and GECKO features with expert-defined concept alignment and agreement-based inference for interpretable, stable report generation.Its MIL aggregator is pretrained using GECKO-derived concept priors, and multiple perturbed inference runs are collated by maximum agreement.Source: §A.3 · p. 20

A.4. ADCT (Fig. S3-(d))

• ADCT maps visual features to structured clinical labels with pathology codebooks and edge-based patch filtering, then generates reports through codebook matching or fine-tuned Flan-T5.It uses CONCH at 20× for all organs except prostate, for which it uses HistoEncoder at 10×.Source: §A.4 · p. 20

A.5. IUCompPath (Fig. S3-(e))

• IUCompPath combines UNI2 and H-Optimus-1 features with a Local-Global Hierarchical encoder and Cross-modal Context module, decoding reports with a 3-layer Transformer.The encoder integrates patch details with slide context, while the CMC module aligns visual and textual embeddings.Source: §A.5 · p. 21

A.6. Team Tiger@REG2025 (Fig. S3-(f))

• Team Tiger adapts WsiCaption with UNI2 features and hierarchical Position-Aware Modules to capture spatial context before Transformer decoding.The team describes this as demonstrating the feasibility of general-purpose pathology foundation models for report generation on diverse Pan-Asia datasets.Source: §A.6 · p. 21

A.7. MedInsight - ViseurAI (Fig. S3-(g))

• MedInsight-ViseurAI uses UNI features with a Transformer decoder aligned to BioGPT, then refines generated reports by semantic similarity to training reports using Sentence-BERT.The contribution describes the UNI encoder for WSI feature extraction; its method specifies fine-tuned UNI-selected patches projected into memory vectors.Source: §A.7 · p. 21

A.8. PathoMozhi (Fig. S3-(h))

• PathoMozhi adapts gated cross-attention to condition a frozen BioGPT language model on CONCH features from 128 sampled patch embeddings per slide.The gated cross-attention layers follow each of the 48 Transformer blocks, and the sampled embeddings serve as visual context tokens.Source: §A.8 · p. 21

A.9. MTS_REG_2025 (Fig. S4-(j))

• MTS REG 2025 retrieves similar historical cases for QEN-3 8B to synthesize reports, using adaptive retrieval and sequential majority voting for report fields.Retrieval stops when the similarity gap between consecutive cases exceeds a threshold; its projection layer is trained with contrastive semantic-similarity supervision.Source: §A.9 · p. 21

A.10. NW-TIA (Fig. S3-(i))

• NW-TIA conditions frozen BioBART on CONCH and Titan slide embeddings, with auxiliary heads predicting organ, specimen type, and 75 findings to ground report generation.The auxiliary predictions regularize the visual projector during training and are removed at inference.Source: §A.10 · p. 21

A.11. REG_Path (Fig. S4-(k))

• REG_Path dynamically adjusts segmentation thresholds to each slide’s statistical distribution to improve feature extraction.Source: §A.11 · p. 21
• Built on SlideChat, REG_Path uses a pretrained CONCH encoder to extract patch-level WSI features with adaptively tuned segmentation thresholds.Source: §A.11 · p. 22
• Its two-stage pipeline aligns domains with 4.2K WSI-VQA pairs, then instruction-tunes on 176K WSI-caption pairs for report generation and clinical coherence.Source: §A.11 · p. 22

B. Challenge Organization Details

• Following BIAS reporting guidelines [46], the paper supplies additional challenge-organization details not fully described in the main manuscript.Source: B. Challenge Organization Details · p. 22

B.1. Challenge Timeline

• Training data were released May 20, 2025; registration deadline and Debug Phase opening were both July 4, 2025, followed by a July 10, 2025 Debug Phase deadline.Source: §B.1 · p. 22
• Test Phase 1 ran July 11–August 1 and Phase 2 August 2–23; winners were announced September 9, 2025.Source: §B.1 · p. 22

B.2. Participation Policy

• Multiple accounts for one participant or institution were prohibited to enforce the two-submission-per-phase limit, and organizers were barred from participating to avoid conflicts of interest.Source: §B.2 · p. 22

B.3. Pre-submission Evaluation

• Teams could make at most two submissions per phase.Source: §B.3 · p. 22
• The organizers publicly released evaluation code for local pre-submission testing instead of providing unrestricted leaderboard access.Source: §B.3 · p. 22
• Submissions that failed the prescribed JSON format, such as by missing case IDs or mismatched key names, were not evaluated.Source: §B.3 · p. 22

B.4. Award Policy

• Awards and prize funding went to the five highest-performing teams, conditional on successfully passing reproducibility verification.Source: §B.4 · p. 22
• Selected teams had to submit source code for independent organizer validation; that code was used internally for reproducibility verification and was not publicly released.Source: §B.4 · p. 22
• The top five teams were invited to present their methods during the Satellite Day 2 session at MICCAI 2025.Source: §B.4 · p. 22

B.5. Data Usage and License Policy

• The anonymized dataset permits attributed non-commercial research, education, method development, benchmarking, and publication; commercial and clinical use and re-identification attempts are prohibited.Redistributed or adapted materials must retain the same license, and data use remains subject to Grand Challenge terms and applicable ethics approvals.Source: §B.5 · p. 22

C. Expertise of Annotators

• The annotation and review team included pathologists spanning over 20 years to under 5 years of experience, plus four non-board-certified pathology trainees.The team comprised three pathologists with more than 20 years’ experience, three with 10–20 years, two with 5–10 years, and one with less than 5 years.Source: C. Expertise of Annotators · p. 22
• Supplementary figures describe diverse team architectures, including hierarchical expert decoding, concept-guided Transformers, retrieval-based synthesis, and prompt-based auxiliary diagnostic heads.The dataset workflow standardized free-text reports, de-identified and quality-checked slides, excluded diagnostically equivocal cases, and paired slides with curated reports.Source: Figure S1 · p. 23, Figure S2 · p. 24, Figure S3 · p. 25
• ICGI achieved full concordance in a breast core biopsy, correctly reporting ductal carcinoma in situ and its type, nuclear grade, necrosis status, and microcalcification.In a separate breast case, ICGI and ICL_PathReport reproduced the diagnosis, Nottingham grade III, and component scores.Source: Table S5 · p. 27, Table S12 · p. 34
• Representative cases showed that reports could capture diagnoses and histologic attributes, but quantitative tumor-volume estimates were often overcalled despite concordant Gleason patterns and scores.The prostate core’s tumor volume was estimated at 50% of core length; reports often estimated 80–95%. ICGI and ICL_PathReport correctly reproduced Nottingham grade III and its component scores in a breast case.Source: Table S6 · p. 28, Table S12 · p. 34
• Most teams identified colonic malignant lymphoma, though some mislabeled the organ; in another stomach lymphoma case, nine teams were correct and two predicted small cell carcinoma.Across teams, most reports identified colon and malignant lymphoma, with occasional stomach organ-label mismatch despite correct diagnosis.Source: Table S7 · p. 29, Table S10 · p. 32
• Squamous morphology led some teams to report lung squamous cell carcinoma for metaplastic carcinoma, a plausible morphologic interpretation but contextually incorrect.ICGI correctly identified metaplastic carcinoma; some other teams gave an indeterminate subtype, “invasive carcinoma of no special type.”Source: Table S8 · p. 30
• For a rare fundic gland polyp represented by only three training cases, MTS_REG_2025 named the entity while most other teams substituted chronic gastritis.Another example showed colorectal-type gland-forming morphology preserved despite incorrect site prediction.Source: Table S9 · p. 31, Table S11 · p. 33