PaperlayerView exact revision on arXiv

Source-linked AI summary

An end-to-end-trained vision-language model for native-language prostate pathology report generation

Christian Grashei, Fabian Gülhan, Maximilian Legnar, Fabian Stögbauer, Cleo-Aron Weis, Carolin Mogler, Peter Schüffler

arXiv:2608.23143v1cs.CV

TL;DR

Pathology departments face a substantial workload from reviewing and reporting each prostate biopsy core, while existing approaches either require report assembly or depend on English-centric language components. This paper develops a native-language, slide-level report-generation framework trained from scratch and evaluates clinical attributes directly, achieving diagnostic performance comparable to classification-based tools and consistent grading improvements across external cohorts.

  • Problem

    Reviewing each prostate biopsy core for malignancy, Gleason grade, and tumor volume creates a substantial pathology workload, while standard NLP metrics poorly capture clinical semantics in generated reports.

  • Method

    The framework trains a tokenizer and vision-language model from scratch, uses a locally deployed LLM to create core-specific WSI-text pairs, and compares generated clinical attributes with ground truth using classification and infiltration metrics.

  • Results

    The end-to-end report-generation model achieves malignancy and grading performance comparable to directly supervised classification, with consistent grading improvements across internal and three external cohorts.

  • Takeaways & Limitations

    Individual institutions can develop native-language report-generation models tailored to their patient populations and reporting languages, with pathologists reviewing and signing off on each draft.

  • Takeaways & Limitations

    Performance is weakest for rare high-grade classes, and predicted tumor infiltration has errors too large for precise clinical measurement.

Abstract

from arXiv · show

Prostate cancer is among the most frequently diagnosed malignancies worldwide, and structured reporting of each biopsy core burdens pathologists. Existing tools frame this as classification, leaving pathologists to assemble coherent reports, while many slide-level vision-language models rely on English-centric encoders that transfer poorly to other clinical languages. We present a slide-level framework generating prostate biopsy reports that is language-independent by construction: tokenizer and model are trained from scratch, demonstrated here in German. To address paired-data scarcity, an automated pipeline uses a locally deployed large language model to split composite reports into core-specific image-text pairs, yielding 17,344 pairs from 2,402 historical cases without manual annotation. Evaluated for clinical attributes rather than linguistic similarity, the model achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, competitive with an FDA-cleared classifier. Grading is further validated on three external cohorts with latent-space augmentation. Institutions can thus train native-language reporting models on their own archives.

Introduction

Prostate pathology reporting requires core-by-core diagnostic assessment, while existing AI tools often classify discrete attributes rather than generate coherent reports. The proposed framework generates native-language reports end to end and uses automated report disaggregation to address paired-data scarcity.

  • Each prostate biopsy core must be reviewed for malignancy, Gleason grade, and tumor volume, creating substantial pathology workload.
  • Existing computational approaches largely frame prostate cancer detection as classification, requiring pathologists to assemble complete reports.
  • Full-WSI vision-language models remain limited by English-centric language encoders that transfer poorly to non-English clinical environments.
  • The framework trains its tokenizer and vision-language model from scratch, generating German reports without translation or pre-trained language components.
  • The automated pipeline converts composite laboratory reports into core-specific WSI-text pairs to support slide-level report generation.

Results

The study builds training pairs by disaggregating composite German pathology reports and addresses language-specific representation through a custom tokenizer. The resulting dataset was audited for extraction accuracy, while English-centric encoders posed challenges for German medical text.

  • A locally deployed large language model extracted core-specific microscopy and critical findings sections within the institutional environment.
  • 17,344 WSI-text pairs were derived from 2,402 cases by disaggregating composite reports into biopsy-specific descriptions.
  • No discrepancies were observed when 200 randomly sampled gradings were manually audited against the original laboratory records.
  • English-centric encoders are poorly suited to non-English clinical environments, creating translation and vocabulary problems for German pathology reports.
  • The custom German pathology tokenizer uses a vocabulary of 10,142 tokens versus BioGPT’s 42,384-token vocabulary.

Reporting of malignancy and grading

The model extracts malignancy and Gleason grading from generated reports and compares them with ground-truth labels. It performs strongly for malignancy and less consistently for grading, with rare high-grade classes presenting a key challenge.

  • The evaluation compares generated-report attributes with extracted malignancy and grading ground truth.
  • 97.6% of core biopsies had correctly detected malignancy, with an F1-score of 96.2%.Specificity was 98.2%, sensitivity 96.4%, and precision 95.9%.
  • 87.2% of biopsies received the correct grade, while macro F1-score was 65.2%.The grading classes included benign, Gleason scores 6, 7a, 7b, 8, and merged scores 9–10.
  • Most grading errors were adjacent-grade misclassifications, consistent with known variability among human pathologists.
  • Gleason scores 6 and 9 had the lowest per-grade recall, with score 9 especially difficult because training examples were scarce.

Reporting tumor infiltration

The model estimates tumor infiltration in either millimeters or percentages, matching the two measurement protocols used in the dataset. Estimates follow the ground truth but become less reliable in underrepresented value ranges.

  • The model reports infiltration as either physical extent in millimeters or percentage, reflecting the dataset’s changing measurement protocol.
  • For extent, mean absolute error was 2.9 mm and Pearson correlation was 0.637; for percentage, mean absolute error was 15.2% and correlation was 0.78.
  • The model distinguished the two infiltration protocols while learning to estimate the reported infiltration concept.
  • The evaluation used 211 extent samples and 82 percentage samples after matching prediction and ground-truth formats.
  • Prediction deviation increased for infiltration ranges that were underrepresented in the data.

Comparison to an ABMIL classification model

The report generation model uses text-guided weak supervision yet performs comparably to directly supervised ABMIL classification, while producing draft reports. Its grading F1-score exceeds ABMIL by 1.7 percentage points, whereas malignancy F1 differs by only 0.2%.

  • Model comparison: The report generation model learns from report text using weak supervision, while ABMIL directly predicts grade probabilities from the same data partitions.This comparison tests whether report text can provide sufficient supervision for clinical attribute prediction.
  • Model comparison: 1.7% higher grading F1-score was achieved by the report generation model than by ABMIL.The authors attribute this to identifying grading information in the training text.
  • Model comparison: 0.2% separated the models’ malignancy F1-scores, with ABMIL showing slightly higher accuracy and precision and report generation higher recall.
  • Model comparison: κ = 0.81 agreement between the two models exceeded their agreement with ground truth, measured at κ = 0.75 and κ = 0.76.The ordering is inconsistent with independent errors, indicating correlated prediction errors.
  • Clinical output: The report generation model produces a draft report in addition to classification attributes, unlike the ABMIL classifier.The FDA-cleared comparison uses an evaluation setting whose exact training dataset is not known.

Comparison to a pan-cancer vision-language model

Compared with the pan-cancer PRISM vision-language model, the prostate-specific report generation model provides more clinically detailed outputs. PRISM achieved strong malignancy classification but substantially weaker grading performance because its outputs often lacked specific grade information.

  • Baseline: PRISM uses a pretrained BioGPT language encoder and was trained on 587,196 slides and 195,344 reports as a pan-cancer baseline.The prostate-specific model is compared with PRISM using the identical Munich evaluation protocol.
  • Clinical detail: PRISM omitted infiltration depth or percentage information, limiting the clinical detail of its abbreviated reports.
  • Performance: 20.3 percentage points separated PRISM’s grading F1-score from the report generation model’s F1-score.PRISM achieved 80.2% accuracy, 53.9% precision, 41.7% recall, and 44.9% F1-score for grading.
  • Clinical detail: 151 samples received no grade or an ambiguous grade from PRISM, including outputs without primary and secondary Gleason pattern breakdowns.These outputs were retained as a separate prediction-only category.
  • Performance: PRISM achieved 96.8% accuracy and 94.5% F1-score for malignancy prediction.

Interpretability

Interpretability is supported by visual attention maps, structured textual details, and embedding visualizations. These outputs help connect predictions to tumor regions, grade composition, and difficult diagnostic contexts.

  • Clinical relevance: Interpretability supports insight into model decision-making and helps resolve discrepancies with pathologist findings.
  • Visual attention: Attention weights mainly correspond to pathologist-annotated tumor regions, including when tumor occupies only a small tissue fraction.This allows the model’s decision to be traced to diagnostically relevant areas.
  • Embedding structure: UMAP embeddings form benign and malignant clusters, with a continuous malignant transition from lower to higher Gleason grades.
  • Textual explanations: Generated reports provide Gleason-grade proportions as an additional learning signal and an explanatory textual output.The proportions can indicate a directional tendency toward higher or lower grades.
  • Textual explanations: The reports can distinguish Gleason scores 3+4 (7a) and 4+3 (7b) and indicate whether immunohistochemistry was performed.These details are especially relevant for borderline or challenging cases.

External cohorts and mitigation of domain shift

The model was evaluated on three external cohorts with differing slide appearances, including substantial saturation and hue shifts. A dual-stage color augmentation pipeline was used to address scanner-induced domain variation.

  • External cohorts: Three external cohorts comprised 424 Heidelberg biopsy cores, 5,160 Radboud biopsies, and 5,456 Karolinska biopsies, none used for training or hyperparameter selection.
  • Domain shift: Karolinska slides were 2.6-fold more saturated than Munich training data and 3.4-fold more saturated than Heidelberg slides.
  • Representation: UMAP embeddings from Munich separated benign and malignant biopsies and showed grade progression within the malignant cluster.
  • Domain shift: Hue shifts placed Radboud toward red and Karolinska toward blue-purple relative to Munich.
  • Mitigation: The latent-space augmentation pipeline rendered each training tile in lighter and darker H&E variants spanning plausible appearance ranges.The variants changed contrast and saturation to address scanner-induced appearance shifts.

Generalization to external cohorts

The report generation model generalized to three external cohorts without fine-tuning or recalibration, maintaining strong malignancy performance while grading remained more difficult. Latent-space augmentation improved grading across every cohort.

  • External evaluation used the same setup as the internal cohort, extracting clinical information from generated reports without fine-tuning or recalibration.The evaluation addressed scanner- and protocol-induced shifts relevant to clinical deployment.
  • 93.0% accuracy, 90.1% recall, 93.7% precision, and 91.9% F1-score were achieved for malignancy prediction on Heidelberg.Grading accuracy was 67.9% with an F1-score of 41.3%.
  • 95.0% and 95.4% F1-scores for malignancy were achieved on Radboud and Karolinska, respectively.Their grading F1-scores were 46.2% and 50.1%.
  • Augmentation improved grading F1-score on every cohort: +8.7 percentage points on Karolinska, +1.6 on Radboud, and +1.5 on Heidelberg.The internal Munich cohort also improved by +4.2 percentage points.
  • Figure 8 summarizes grade distributions, malignancy and Gleason confusion matrices, and augmentation-related macro F1-score increases across the external cohorts.

Discussion

The study argues that end-to-end report generation can preserve clinically relevant diagnostic performance while producing native-language drafts for pathologist review. Its main boundaries include weaker rare-grade recognition, imprecise infiltration estimates, retrospective evaluation, and risks requiring verification.

  • End-to-end report generation from scratch achieved internal malignancy and grading performance comparable to label-trained classification.The result indicates that generating free text did not compromise diagnostic accuracy on the internal cohort.
  • Dual deterministic color and latent-space augmentation yielded consistent grading improvements across the internal and three external cohorts.The strategy may transfer to foundation-model embedding pipelines, particularly in low-data settings.
  • Adjacent-grade errors predominated and mirrored documented inter-observer variability, but propagation of this variability through training and evaluation remains unresolved.The study also notes that realistic color augmentation needs more systematic evaluation and increases computational and storage requirements.
  • The model can be tailored to an institution’s patient population and reporting language using its own data.The discussion presents this as feasible despite a smaller and uncurated training corpus than the independently developed FDA-cleared tool.
  • Native-language draft reports can support a human-in-the-loop workflow in which pathologists review, correct, and sign off while retaining diagnostic responsibility.Deterministic greedy decoding makes reports reproducible and auditable, but fluent incorrect grades or infiltration values remain possible.
  • The model performed weakest on rare high-grade classes, especially Gleason score ≥9, because few training examples limited recognition of their morphologic patterns.Predicted infiltration was also too imprecise for clinical measurement, with mean absolute errors of 2.9 mm and 15.2% and R2 scores below 0.5.
  • The evaluation was retrospective and based entirely on archival data, leaving prospective or reader-study validation necessary for clinical deployment and workload assessment.

Methods

The study constructs core-level WSI–text data from retrospective pathology records and trains a slide-level report-generation model with clinical-attribute evaluation. The workflow combines whole-slide processing, native-language text modeling, spatially informed attention, and structured assessment of generated reports.

  • Data and cohorts: 17,344 WSIs from 2,402 cases formed the Munich dataset, with additional Heidelberg, Karolinska, and Radboud cohorts used for evaluation or comparison.The Munich, Heidelberg, Karolinska, and Radboud cohorts differ in case composition, scanners, and resolution.
  • Data processing: A locally deployed LLM disaggregated composite case reports into microscopy and critical-findings text paired with the corresponding biopsy WSI.Reports were segmented into primary sections, then filtered by specimen identifiers before pairing text with each biopsy image.
  • Image processing: WSIs were segmented and tessellated into non-overlapping 512×512-pixel tissue patches before multimodal modeling.TRIDENT and GrandQC were used to isolate tissue from background and pen marks.
  • Model architecture: The report generator combines a slide encoder, text encoder, and multimodal decoder, with a frozen UNI2-h patch embedding projected into the slide encoder.The slide encoder uses transformer blocks and a trainable CLS token to represent the slide.
  • Interpretability: Attention rollout recursively aggregates transformer attention across blocks to assign patch-level values that highlight influential tissue regions.The resulting values are combined with original patch coordinates for visualization.
  • Evaluation: Structured labels captured malignancy, Gleason grade, and infiltration for evaluating diagnostic correctness rather than linguistic similarity.The extracted categorical and continuous attributes correspond to clinical reporting requirements and were compared with ground truth.

Supplementary Material

The supplementary material documents the prompts used to extract report sections, specimen-level text, and Gleason scores, alongside a color-variation augmentation strategy. These procedures support automated preparation of structured pathology inputs and training images.

  • Report section extraction: The section-extraction prompt requests Clinical Information/Question, Macroscopy, Microscopy, and Critical Report under fixed headings.Its output is designed for automatic parsing and excludes introductory or closing remarks.
  • Specimen-level extraction: Specimen-level extraction prompts isolate all text referring to a specified tissue block, including references to multiple block numbers.If no relevant description exists, the prompt requires the exact response “No information.”
  • Label extraction: The Gleason-score prompt returns only the score, “no carcinoma,” or “No information,” depending on the report content.This creates a constrained output for extracting a key clinical label.
  • Color augmentation: Each training tissue patch receives two additional color variants: a lighter version and a darker version.The variants reduce or increase contrast and saturation to simulate visual variability from scanners and staining protocols.
Loading 2608.23143v1…