An end-to-end-trained vision-language model for native-language prostate pathology report generation

• Each prostate biopsy core must be reviewed for malignancy, Gleason grade, and tumor volume, creating substantial pathology workload.Source: Introduction · p. 1
• Existing computational approaches largely frame prostate cancer detection as classification, requiring pathologists to assemble complete reports.Source: Introduction · p. 1
• Full-WSI vision-language models remain limited by English-centric language encoders that transfer poorly to non-English clinical environments.Source: Introduction · p. 1
• The framework trains its tokenizer and vision-language model from scratch, generating German reports without translation or pre-trained language components.Source: Introduction · pp. 1–2
• The automated pipeline converts composite laboratory reports into core-specific WSI-text pairs to support slide-level report generation.Source: Introduction · p. 2

• A locally deployed large language model extracted core-specific microscopy and critical findings sections within the institutional environment.Source: An automatically curated dataset of… · p. 2
• 17,344 WSI-text pairs were derived from 2,402 cases by disaggregating composite reports into biopsy-specific descriptions.Source: An automatically curated dataset of… · p. 2
• No discrepancies were observed when 200 randomly sampled gradings were manually audited against the original laboratory records.Source: An automatically curated dataset of… · p. 2
• English-centric encoders are poorly suited to non-English clinical environments, creating translation and vocabulary problems for German pathology reports.Source: Domain-specific tokenization and… · p. 2
• The custom German pathology tokenizer uses a vocabulary of 10,142 tokens versus BioGPT’s 42,384-token vocabulary.Source: Domain-specific tokenization and… · p. 2

• The evaluation compares generated-report attributes with extracted malignancy and grading ground truth.Source: Reporting of malignancy and grading · p. 2
• 97.6% of core biopsies had correctly detected malignancy, with an F1-score of 96.2%.Specificity was 98.2%, sensitivity 96.4%, and precision 95.9%.Source: Reporting of malignancy and grading · p. 3
• 87.2% of biopsies received the correct grade, while macro F1-score was 65.2%.The grading classes included benign, Gleason scores 6, 7a, 7b, 8, and merged scores 9–10.Source: Reporting of malignancy and grading · p. 3
• Most grading errors were adjacent-grade misclassifications, consistent with known variability among human pathologists.Source: Reporting of malignancy and grading · p. 3
• Gleason scores 6 and 9 had the lowest per-grade recall, with score 9 especially difficult because training examples were scarce.Source: Reporting of malignancy and grading · p. 4

• The model reports infiltration as either physical extent in millimeters or percentage, reflecting the dataset’s changing measurement protocol.Source: Reporting tumor infiltration · p. 4
• For extent, mean absolute error was 2.9 mm and Pearson correlation was 0.637; for percentage, mean absolute error was 15.2% and correlation was 0.78.Source: Reporting tumor infiltration · p. 4
• The model distinguished the two infiltration protocols while learning to estimate the reported infiltration concept.Source: Reporting tumor infiltration · p. 4
• The evaluation used 211 extent samples and 82 percentage samples after matching prediction and ground-truth formats.Source: Reporting tumor infiltration · p. 4
• Prediction deviation increased for infiltration ranges that were underrepresented in the data.Source: Figure 3 · p. 5

Model comparison

• The report generation model learns from report text using weak supervision, while ABMIL directly predicts grade probabilities from the same data partitions.This comparison tests whether report text can provide sufficient supervision for clinical attribute prediction.Source: Comparison to an ABMIL classification… · p. 5
• 1.7% higher grading F1-score was achieved by the report generation model than by ABMIL.The authors attribute this to identifying grading information in the training text.Source: Comparison to an ABMIL classification… · p. 5
• 0.2% separated the models’ malignancy F1-scores, with ABMIL showing slightly higher accuracy and precision and report generation higher recall.Source: Comparison to an ABMIL classification… · p. 5
• κ = 0.81 agreement between the two models exceeded their agreement with ground truth, measured at κ = 0.75 and κ = 0.76.The ordering is inconsistent with independent errors, indicating correlated prediction errors.Source: Comparison to an ABMIL classification… · p. 5

Clinical output

• The report generation model produces a draft report in addition to classification attributes, unlike the ABMIL classifier.The FDA-cleared comparison uses an evaluation setting whose exact training dataset is not known.Source: Figure 4 · p. 6, Comparison to an FDA-cleared… · p. 7

Baseline

• PRISM uses a pretrained BioGPT language encoder and was trained on 587,196 slides and 195,344 reports as a pan-cancer baseline.The prostate-specific model is compared with PRISM using the identical Munich evaluation protocol.Source: Comparison to a pan-cancer… · p. 7

Clinical detail

• PRISM omitted infiltration depth or percentage information, limiting the clinical detail of its abbreviated reports.Source: Comparison to a pan-cancer… · p. 7

Performance

• 20.3 percentage points separated PRISM’s grading F1-score from the report generation model’s F1-score.PRISM achieved 80.2% accuracy, 53.9% precision, 41.7% recall, and 44.9% F1-score for grading.Source: Comparison to a pan-cancer… · p. 7

Clinical detail

• 151 samples received no grade or an ambiguous grade from PRISM, including outputs without primary and secondary Gleason pattern breakdowns.These outputs were retained as a separate prediction-only category.Source: Comparison to a pan-cancer… · p. 7

Performance

• PRISM achieved 96.8% accuracy and 94.5% F1-score for malignancy prediction.Source: Comparison to a pan-cancer… · p. 7

Clinical relevance

• Interpretability supports insight into model decision-making and helps resolve discrepancies with pathologist findings.Source: Interpretability · p. 7

Visual attention

• Attention weights mainly correspond to pathologist-annotated tumor regions, including when tumor occupies only a small tissue fraction.This allows the model’s decision to be traced to diagnostically relevant areas.Source: Interpretability · p. 7, Figure 5 · p. 8

Embedding structure

• UMAP embeddings form benign and malignant clusters, with a continuous malignant transition from lower to higher Gleason grades.Source: Interpretability · p. 7

Textual explanations

• Generated reports provide Gleason-grade proportions as an additional learning signal and an explanatory textual output.The proportions can indicate a directional tendency toward higher or lower grades.Source: Interpretability · pp. 7, 9, Figure 6 · p. 9
• The reports can distinguish Gleason scores 3+4 (7a) and 4+3 (7b) and indicate whether immunohistochemistry was performed.These details are especially relevant for borderline or challenging cases.Source: Interpretability · p. 9, Figure 6 · p. 9

External cohorts

• Three external cohorts comprised 424 Heidelberg biopsy cores, 5,160 Radboud biopsies, and 5,456 Karolinska biopsies, none used for training or hyperparameter selection.Source: External cohorts and mitigation of… · p. 9

Domain shift

• Karolinska slides were 2.6-fold more saturated than Munich training data and 3.4-fold more saturated than Heidelberg slides.Source: External cohorts and mitigation of… · p. 9

Representation

• UMAP embeddings from Munich separated benign and malignant biopsies and showed grade progression within the malignant cluster.Source: Figure 7 · p. 10

Domain shift

• Hue shifts placed Radboud toward red and Karolinska toward blue-purple relative to Munich.Source: External cohorts and mitigation of… · p. 10

Mitigation

• The latent-space augmentation pipeline rendered each training tile in lighter and darker H&E variants spanning plausible appearance ranges.The variants changed contrast and saturation to address scanner-induced appearance shifts.Source: External cohorts and mitigation of… · p. 10

• External evaluation used the same setup as the internal cohort, extracting clinical information from generated reports without fine-tuning or recalibration.The evaluation addressed scanner- and protocol-induced shifts relevant to clinical deployment.Source: Generalization to external cohorts · p. 10
• 93.0% accuracy, 90.1% recall, 93.7% precision, and 91.9% F1-score were achieved for malignancy prediction on Heidelberg.Grading accuracy was 67.9% with an F1-score of 41.3%.Source: Generalization to external cohorts · p. 10
• 95.0% and 95.4% F1-scores for malignancy were achieved on Radboud and Karolinska, respectively.Their grading F1-scores were 46.2% and 50.1%.Source: Generalization to external cohorts · p. 10
• Augmentation improved grading F1-score on every cohort: +8.7 percentage points on Karolinska, +1.6 on Radboud, and +1.5 on Heidelberg.The internal Munich cohort also improved by +4.2 percentage points.Source: Generalization to external cohorts · p. 10
• Figure 8 summarizes grade distributions, malignancy and Gleason confusion matrices, and augmentation-related macro F1-score increases across the external cohorts.Source: Figure 8 · p. 11

• End-to-end report generation from scratch achieved internal malignancy and grading performance comparable to label-trained classification.The result indicates that generating free text did not compromise diagnostic accuracy on the internal cohort.Source: Discussion · p. 12
• Dual deterministic color and latent-space augmentation yielded consistent grading improvements across the internal and three external cohorts.The strategy may transfer to foundation-model embedding pipelines, particularly in low-data settings.Source: Discussion · p. 12
• Adjacent-grade errors predominated and mirrored documented inter-observer variability, but propagation of this variability through training and evaluation remains unresolved.The study also notes that realistic color augmentation needs more systematic evaluation and increases computational and storage requirements.Source: Discussion · p. 12
• The model can be tailored to an institution’s patient population and reporting language using its own data.The discussion presents this as feasible despite a smaller and uncurated training corpus than the independently developed FDA-cleared tool.Source: Discussion · p. 12
• Native-language draft reports can support a human-in-the-loop workflow in which pathologists review, correct, and sign off while retaining diagnostic responsibility.Deterministic greedy decoding makes reports reproducible and auditable, but fluent incorrect grades or infiltration values remain possible.Source: Discussion · p. 12
• The model performed weakest on rare high-grade classes, especially Gleason score ≥9, because few training examples limited recognition of their morphologic patterns.Predicted infiltration was also too imprecise for clinical measurement, with mean absolute errors of 2.9 mm and 15.2% and R2 scores below 0.5.Source: Discussion · p. 12
• The evaluation was retrospective and based entirely on archival data, leaving prospective or reader-study validation necessary for clinical deployment and workload assessment.Source: Discussion · p. 12

Data and cohorts

• 17,344 WSIs from 2,402 cases formed the Munich dataset, with additional Heidelberg, Karolinska, and Radboud cohorts used for evaluation or comparison.The Munich, Heidelberg, Karolinska, and Radboud cohorts differ in case composition, scanners, and resolution.Source: Munich cohort · p. 13, Heidelberg cohort · p. 13, PANDA (Karolinska and Radboud cohort) · p. 13

Data processing

• A locally deployed LLM disaggregated composite case reports into microscopy and critical-findings text paired with the corresponding biopsy WSI.Reports were segmented into primary sections, then filtered by specimen identifiers before pairing text with each biopsy image.Source: Creation of WSI-text pairs and… · p. 13

Image processing

• WSIs were segmented and tessellated into non-overlapping 512×512-pixel tissue patches before multimodal modeling.TRIDENT and GrandQC were used to isolate tissue from background and pen marks.Source: WSI preprocessing · p. 14

Model architecture

• The report generator combines a slide encoder, text encoder, and multimodal decoder, with a frozen UNI2-h patch embedding projected into the slide encoder.The slide encoder uses transformer blocks and a trainable CLS token to represent the slide.Source: Model architecture · p. 14

Interpretability

• Attention rollout recursively aggregates transformer attention across blocks to assign patch-level values that highlight influential tissue regions.The resulting values are combined with original patch coordinates for visualization.Source: Attention maps · p. 16

Evaluation

• Structured labels captured malignancy, Gleason grade, and infiltration for evaluating diagnostic correctness rather than linguistic similarity.The extracted categorical and continuous attributes correspond to clinical reporting requirements and were compared with ground truth.Source: Creation of WSI-text pairs and… · p. 13, Evaluation metrics · p. 16

Report section extraction

• The section-extraction prompt requests Clinical Information/Question, Macroscopy, Microscopy, and Critical Report under fixed headings.Its output is designed for automatic parsing and excludes introductory or closing remarks.Source: Supplementary Material · p. 20

Specimen-level extraction

• Specimen-level extraction prompts isolate all text referring to a specified tissue block, including references to multiple block numbers.If no relevant description exists, the prompt requires the exact response “No information.”Source: Supplementary Material · p. 20

Label extraction

• The Gleason-score prompt returns only the score, “no carcinoma,” or “No information,” depending on the report content.This creates a constrained output for extracting a key clinical label.Source: Supplementary Material · p. 20

Color augmentation

• Each training tissue patch receives two additional color variants: a lighter version and a darker version.The variants reduce or increase contrast and saturation to simulate visual variability from scanners and staining protocols.Source: Figure S1 · p. 21