Source-linked AI summary
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
Yuxuan Yang, Zhonghao Yan, Yi Zhang, Bo Yun, Muxi Diao, Guowei Zhao, Kongming Liang, Wenbin Li, Zhanyu Ma
TL;DR
HCC diagnosis from gigapixel WSIs requires representations that retain diagnostic detail without excessive redundancy and support multiple tissue scales. Hepato-LLaVA combines Sparse Topo-Pack Attention, HepatoPathoVQA, and staged alignment, achieving state-of-the-art benchmark performance with an average score of 0.83.
Problem
Gigapixel WSI analysis is constrained by information loss in thumbnail representations, redundant aggregation in slide encoders, and limited support for patch-level and multi-scale diagnosis.
Method
Hepato-LLaVA uses 2D topology-aware Sparse Topo-Pack Attention, a multi-scale HepatoPathoVQA dataset, and LoRA fine-tuning with a hierarchical slide representation.
Results
Avg 0.83 on HepatoPathoBench, leading the best WSI-based runner-up SlideChat at 0.66 and outperforming thumbnail-based models at Avg 0.50–0.57.
Takeaways & Limitations
Topology-aware representations outperform high-dimensional redundant features for efficient hepatocellular pathology analysis.
Abstract
from arXiv · showhide
Hepatocellular Carcinoma diagnosis relies heavily on the interpretation of gigapixel Whole Slide Images. However, current computational approaches are constrained by fixed-resolution processing mechanisms and inefficient feature aggregation, which inevitably lead to either severe information loss or high feature redundancy. To address these challenges, we propose Hepato-LLaVA, a specialized Multi-modal Large Language Model designed for fine-grained hepatocellular pathology analysis. We introduce a novel Sparse Topo-Pack Attention mechanism that explicitly models 2D tissue topology. This mechanism effectively aggregates local diagnostic evidence into semantic summary tokens while preserving global context. Furthermore, to overcome the lack of multi-scale data, we present HepatoPathoVQA, a clinically grounded dataset comprising 33K hierarchically structured question-answer pairs validated by expert pathologists. Our experiments demonstrate that Hepato-LLaVA achieves state-of-the-art performance on HCC diagnosis and captioning tasks, significantly outperforming existing methods. Our code and implementation details are available at https://pris-cv.github.io/Hepto-LLaVA/.
1 Introduction
HCC pathology analysis is difficult because gigapixel WSIs are heterogeneous, labor-intensive, and poorly served by fixed-resolution or globally aggregated representations. Hepato-LLaVA addresses these gaps with topology-aware slide encoding and a clinically grounded multi-scale VQA dataset.
- Gigapixel WSI interpretation is labor-intensive, and subtle early-stage lesions are prone to inter-observer variability.
- Thumbnail methods lose patch-level details, while slide-encoder methods aggregate patches globally and remain less equipped for patch-level diagnosis.
- The paper investigates how to preserve critical diagnostic details while minimizing redundancy and how to support variable-resolution, multi-scale diagnosis.
- Sparse Topo-Pack Attention models 2D tissue topology, reduces spatial redundancy, and condenses features into fixed-length query tokens.
- HepatoPathoVQA contains over 33K expert-validated QA pairs spanning WSI, ROI, and patch scales across the clinical workflow.
- Hepato-LLaVA achieves a 20% improvement in average diagnostic accuracy over existing open-source pathology MLLMs.
2 Methods
The method constructs multi-scale pathology inputs, organizes patch features into hierarchical packs, and applies sparse topology-aware attention to aggregate local evidence while retaining global context. A staged training strategy and QFormer connector then align these representations with the language model.
- Patch Encoding: WSIs are tessellated into non-overlapping P × P patches, which a frozen encoder maps into a D-dimensional spatial feature grid.
- Dataset Construction: HepatoPathoVQA uses MST-based clustering and triangular seed selection to form ROIs, then generates hierarchical multi-scale QA pairs and captions.
- Slide Encoding: The slide encoder processes flattened patch sequences because directly using thousands of patch features would create excessive dimensionality and redundancy.
- Sparse Topo-Pack Attention: Structured input groups patches into row-major k × k Summary Packs, each with a semantic summary token derived from its local image region.
- Sparse Topo-Pack Attention: The sparse mask supports Global Sink, Intra-Pack, aggregation, and summary-level interactions to combine local evidence with long-range context.
- Sparse Topo-Pack Attention: With k = 3, the hierarchical attention reduces attention overhead to ≈1% of the dense counterpart.
- Training: Training uses MAE pre-training, MoCo pre-training, and LoRA-based instruction tuning, with a QFormer connector for variable-length summary-token sequences.
3 Experiments
Experiments evaluate Hepato-LLaVA on a disjoint multi-scale benchmark using clinical-response metrics and ablations of connectors, training, and token counts. The model achieves strong overall and scale-consistent performance, while sparse token condensation and Q-Former training improve efficiency and accuracy.
- Experimental Settings: HepatoPathoBench contains 3K evaluation pairs disjoint from training, spanning WSI, ROI, and Patch scales.Evaluation uses METEOR and WSI-P for open-ended responses, strict Accuracy for single-choice questions, and partial credit for incomplete multiple-choice answers.
- Main Results: Hepato-LLaVA achieves Avg 0.83, leading the best WSI-based runner-up SlideChat by 0.17.Thumbnail-based models score Avg 0.50–0.57, while Hepato-LLaVA also reaches WSI-P 0.79 in morphology and 0.75 in diagnosis.
- Main Results: Hepato-LLaVA maintains robust performance across WSI 0.82, ROI 0.83, and Patch 0.83 scales.These scores exceed the WSI-LLaVA backbone’s corresponding 0.65, 0.67, and 0.64 results.
- Ablation Study: The Q-Former connector outperforms the MLP baseline by +5.18% on WSI and has lower variability, with std = 0.26 versus 1.77.The Two-Stage setting adds +2.76% on ROI compared with direct fine-tuning.
- Ablation Study: Using 32 learnable Q-Former queries yields peak accuracy of 88.75% on ROI and 88.10% on Patch, whereas increasing token counts introduces noise and degrades performance.The selected 32-query configuration balances inference efficiency with robust performance across spatial scales.
4 Conclusion
Hepato-LLaVA combines topology-aware slide representations with a multi-scale HCC VQA dataset for hepatocellular pathology analysis. The authors report state-of-the-art performance and argue that embedding pathological priors supports efficient precision pathology AI.
- 4 Conclusion: Extensive experiments demonstrate state-of-the-art performance and report that topology-aware representations outperform high-dimensional redundant features.The conclusion frames embedding pathological priors into deep learning as a direction for efficient AI in precision pathology.