PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

1. Introduction

• Existing MLLMs provide little transparent reasoning, while model-contained knowledge is tied to model scale and costly to update.Prior reports may provide only answers or captions, and knowledge in model weights requires thorough fine-tuning to update.Source: §1 · pp. 1–2
• PathKGen extracts slide features with pathology foundation models, queries a UMLS-derived pathology graph for morphology, and supplies visual and knowledge tokens to an MLLM.Its retrieved subgraph is shown to users as an intermediate result.Source: §1 · p. 2, Figure 2 · p. 3
• The authors describe knowledge-graph curation from credible sources and suggest extending the approach to other diagnostic fields through domain-specific graphs.They characterize pathology knowledge-graph use for human-interpretable report generation as underexplored.Source: §1 · p. 2
• PathKGen's evaluation is reported to surpass state-of-the-art baselines across all evaluated metrics while improving explainability.The authors also describe the method as the first to incorporate knowledge graphs for human-interpretable pathology report generation.Source: §1 · p. 2

2.1. MLLMs for WSI Analysis

• WSI report-generation systems commonly project slide-level features into an MLLM’s semantic space and combine visual tokens with prompts to generate reports.The described routine aggregates patch features with a slide encoder before projection.Source: §2.1 · p. 3

2.2. Domain Knowledge Integrated WSI Analysis

• Pathology knowledge methods include graph-guided VQA, concept alignment for few-shot classification, and task-specific graph pooling across pathology tasks.K-PathVQA’s graph is encoded as dense embeddings, which the authors say removes human readability and transparency.Source: §2.2 · p. 3
• The authors identify structured medical knowledge for pathology report generation as unexplored and use a pathology-specific graph for explicit, human-interpretable reasoning pathways.The prior methods discussed primarily target classification or representation learning.Source: §2.2 · p. 3

3. PathKGen

• The pipeline processes each WSI through parallel visual and knowledge paths, then combines their tokens with an instruction prompt for report generation.The visual path extracts slide features; the knowledge path predicts histology and retrieves a graph substructure.Source: §3 · pp. 3–4

3.1. Visual Path

• CONCHv1.5 extracts patch features and TITAN aggregates them into slide tokens, which a two-layer MLP projects into the MLLM’s semantic space.Attention pooling of slide tokens also supplies 768-dimensional embeddings to train the auxiliary classifier.Source: §3.1 · p. 4

3.2. Knowledge Path

• The auxiliary classifier combines slide embeddings with primary-site metadata; together they achieve 0.9711 Macro-F1, compared with 0.8754 for slide embeddings alone.The classifier is trained separately before the MLLM; primary site alone achieves 0.6941 Macro-F1.Source: §3.2 · pp. 4–5
• The knowledge path predicts histology from slide embeddings and primary site, then retrieves the predicted class’s one-hop morphology subgraph as structured, tokenized knowledge.The subgraph’s six relation categories organize morphological features, and the graph is visible to users; slides assigned the same histology receive identical knowledge tokens.Source: §3.2 · pp. 4–5

3.3. Language Model

• The MLLM is trained in two stages: projection-layer alignment with the backbone frozen, followed by QLoRA-based visual instruction tuning with the MLLM unfrozen.The unified prompt contains visual tokens, knowledge tokens, and an instruction; classifier confidence is stated so the MLLM does not rely solely on its prediction.Source: §3.3 · pp. 5–6

3.4. Knowledge Graph Curation

• The knowledge graph is built from report-derived morphological features mapped to UMLS concepts, assigned semantic relations, and verified by pathologists.It contains 177 nodes and 317 edges across six relation categories, with graph density 0.01 and average node outdegree 1.79.Source: §3.4 · p. 6

3.5. Dataset and Training.

• The report-generation pipeline uses WSI-Bench, with 9,749 training pairs and 208 test pairs; 9,452 slides remain after excluding slides without microns-per-pixel values.Histology labels are derived from reports and cross-validated against TCGA portal labels; primary-site metadata also comes from TCGA.Source: §3.5 · p. 7

4.1. Quantitative Results on WSI-Bench Dataset.

• BLEU-1/2/3/4 improved 6.67%/8.66%/10.10%/8.75%, ROUGE-L 5.92%, METEOR 2.75%, and WSI-Precision reached 0.480, 9.09% above SOTA.The WSI-Precision metric verifies generated predictions against ground-truth claims.Source: §4.1 · pp. 7–8

4.2. Evaluation on Semantic Alignment

• BERTScore F1 was 0.9081 with RoBERTa-large and 0.8836 with DistilBERT; WSI-LLaVA scored 0.9068 with RoBERTa-large.The authors interpret the high scores as strong semantic alignment and minimal factual distortion.Source: §4.2 · p. 8

4.3. Qualitative Results Interpretation

• The knowledge-path descriptors overlap with the report, while the MLLM also learns graph-absent connections and negation rules from absent morphological features.In Fig. 5(a), the subgraph and report descriptors overlap; Fig. 5(b–c) illustrate novel connections and absent-feature evidence.Source: §4.3 · p. 8

4.4. Ablation Studies

• At 7B parameters, PathKGen improves BLEU-4 by 2.47%, METEOR by 2.52%, and WSI-Precision by 3.03% over the vision-only baseline.This ablation tests the knowledge path’s contribution at a fixed model scale.Source: §4.4 · p. 8
• PathKGen outperforms variants conditioned only on predicted histology or primary site, supporting the value of structured relations and morphological features.The richer knowledge provides complementary information beyond histology or site conditioning alone.Source: §4.4 · p. 8
• At inference, oracle knowledge slightly improves performance, whereas random incorrect histology anchors substantially degrade WSI-Precision.Table 5 describes these as counterfactual interventions on a fixed trained model to assess sensitivity to retrieved knowledge.Source: §4.4 · p. 8, Table 5 · p. 8
• Lower-parameter models achieve performance comparable to 7B SOTA with the same knowledge-path architecture, though scaling the model yields greater gains.The authors characterize visual discrimination logic and knowledge containment as partially externalized to the knowledge path.Source: §4.4 · p. 8

5. Conclusion

• PathKGen integrates a curated pathology knowledge graph into a standard MLLM architecture for report generation with human-readable intermediate results.The conclusion describes this as outsourcing knowledge containment and calls it, to the authors’ knowledge, the first method to do so.Source: §5 · p. 8
• Benchmark experiments show superior performance over existing models, and ablations support the effectiveness of adding knowledge.Source: §5 · p. 8
• Generalizing the method to other diagnostic routines is left for future work.Source: §5 · p. 8

A. Analysis of Auxiliary Classifier.

• Table 6 reports per-class histology classification metrics for the auxiliary classifier.Source: Table 6 · p. 11
• The reported classification metrics are evaluated on the test set.Source: Table 6 · p. 11
• The table concerns histology classification by the auxiliary classifier, not report-generation metrics.Source: Table 6 · p. 11

B. Training Details.

• WSIs are tiled into 512 × 512 patches at 20× magnification after tissue segmentation, and tissue-free patches are discarded.Filtered patches are encoded with CONCH v1.5 and aggregated into slide representations using TITAN.Source: B. Training Details · p. 11
• Training first fits the projection layer with the LLM frozen for 8 epochs, then fine-tunes the projection layer and LLM with QLoRA for one epoch.Stage 1 uses an effective batch size of 64; stage 2 uses 128. The passage also specifies separate learning rates for QLoRA-adapted LLM parameters and the projection layer.Source: B. Training Details · p. 11
• Figure 6 presents the auxiliary classifier’s confusion matrix, evaluated on the test set.Source: Figure 6 · p. 12
• The random seed for functions from every Python class is 42.Source: B. Training Details · p. 12

• The training objective is label-smoothed per-token autoregressive cross-entropy computed only over the report segment.Label smoothing uses ϵ = 0.1.Source: §B.1 · p. 12
• At periodic steps, BLEU-4 scores against reference reports provide a hard-example signal for reweighting each sample’s loss.The model greedily generates a report for each sample before scoring it.Source: §B.1 · p. 12
• The weight signal is measured relative to a running baseline β, initialized to 0.2, with the mined micro-batch’s mean BLEU-4 denoted by b̄.Source: §B.1 · p. 12

C. Multiple Run Statistics.

• Confidence intervals over repeated evaluation runs indicate stable PathKGen metrics and suggest differences across knowledge-path conditions are unlikely to reflect run-to-run variance alone.The reported metrics include BLEU-1/2/3/4, ROUGE-L, METEOR, and WSI-Precision.Source: Table 7 · p. 13

D. Prompt Template.

• The prompt uses system, user, and assistant roles, with TITAN visual features placed between reserved image-start and image-end markers.Source: D. Prompt Template · p. 13
• Training supervises only assistant-turn report tokens, masking the preceding sequence so the model conditions on knowledge and the question.Source: D. Prompt Template · p. 13
• The assistant turn contains the pathologist-written report during training, while inference starts autoregressive next-token generation from that turn.The system prompt casts the model as a pathology assistant asked for detailed, accurate diagnostic observations based on microscopic features.Source: Figure 7 · p. 13, D. Prompt Template · p. 13

• In the graph visualizations, green marks evidence present in the WSI, red marks absent evidence, and blue marks predicted histology.Present and absent features are highlighted through rule-based post-hoc text processing.Source: E. Qualitative Analysis · p. 13, Figure 8 · p. 14
• PathKGen captures more semantics than WSI-LLaVA in qualitative report comparisons against ground truth.The comparisons cover sample WSIs in Tables 8, 9, and 10.Source: E. Qualitative Analysis · p. 13, Table 8 · p. 15, Table 9 · p. 16
• The report comparison annotations mark ground-truth findings in green, hallucinated or contradictory findings in red, omitted findings in orange, and template language with underlining.Source: E. Qualitative Analysis · p. 15