Source-linked AI summary
TC-SSA: Token Compression via Semantic Slot Aggregation for Gigapixel Pathology Reasoning
Zhuo Chen, Shawn Young, Lijian Xu
TL;DR
Gigapixel WSIs create patch sequences beyond standard Transformer limits, motivating efficient pathology VLM processing. TC-SSA compresses patch features into semantic slots through sparse routing and aggregation, achieving strong benchmark and MIL classification results under constrained token budgets.
Problem
Whole-slide images can contain more than 10^5 patches, making direct processing exceed standard Transformer limits and motivating compression that preserves global diagnostic context.
Method
TC-SSA uses learnable semantic slots, Top-2 gated routing, weighted aggregation, and regularization to compress all patch features into a fixed token budget.
Results
TC-SSA achieves 78.34% overall accuracy on SlideBench(TCGA), 77.14% on Diagnosis, and AUCs of 95.83% on TCGA-BRCA and 98.27% on TCGA-NSCLC.
Takeaways & Limitations
Learnable semantic aggregation provides strong performance under constrained visual-token budgets and generalizes across pathology reasoning and classification settings.
Abstract
from arXiv · showhide
The application of large vision-language models to computational pathology holds great promise for diagnostic assistants but faces a critical computational bottleneck: the gigapixel scale of Whole Slide Images (WSIs). A single WSI typically contains over 105 patches, creating sequence lengths that exceed the constraints of standard Transformer architectures. Existing solutions often resort to spatial sampling, which risks discarding diagnostically critical evidence. To address this, we propose TC-SSA (Token Compression via Semantic Slot Aggregation), a learnable token compression framework that aggregates patch features into a fixed number of semantic slots. A gated routing module assigns patches to slots using sparse Top-2 routing, followed by weighted aggregation, enabling global slide coverage under a strict token budget. The resulting representation retains diagnostically relevant information while reducing the number of visual tokens to 1.7% of the original sequence. On the SlideBench(TCGA), our model achieves 78.34% overall accuracy and 77.14% on the diagnosis subset, outperforming sampling-based baselines under comparable token budgets. The method also generalizes to MIL classification, reaching AUC of 95.83% on TCGA-BRCA, 98.27% on TCGA-NSCLC and 79.80% on PANDA. These results suggest that learnable semantic aggregation provides an effective trade-off between efficiency and diagnostic performance for gigapixel pathology reasoning.
1 Introduction
TC-SSA addresses the scalability bottleneck of gigapixel pathology slides with learnable semantic slot aggregation, preserving global context under a strict token budget. It combines semantic routing, regularization, and fixed-budget compression to retain critical evidence while reducing computational demands.
- A single WSI contains more than 10^5 patches, exceeding standard Transformer memory and computation limits for direct processing.
- TC-SSA aggregates all patch features into fixed learnable semantic slots to compress visual tokens while maintaining global slide coverage.
- Semantic slot routing uses shared contextual relevance rather than spatial proximity, aggregating sparse diagnostic evidence and suppressing redundant background.
- Semantic affinity clustering, load balancing, entropy regularization, and z-loss mitigate slot collapse, stabilize routing, and encourage balanced slot utilization.
- 1.7% visual-token usage yields 78.34% overall accuracy on SlideBench(TCGA) and 77.14% on its Diagnosis subset.
2 Methodology
TC-SSA compresses variable-length WSI patch features into a fixed semantic-slot representation through sparse gated routing, weighted aggregation, and auxiliary routing regularization. The method targets efficient preservation of global diagnostic context under a strict token budget.
- Problem formulation: The compression function maps X ∈ R^B×N×D to X′ ∈ R^B×K×D, with K ≪ N, to fit dense slide evidence within the VLM context window.N routinely exceeds 10^5 patches for a single gigapixel slide.
- Semantic slot aggregation: TC-SSA uses a lightweight gate to assign each patch to semantic slots, retaining only its two highest-probability routes.The truncated routing weights are formed by masking the gate probabilities with a Top-2 selection.
- Semantic slot aggregation: Weighted pooling consolidates spatially dispersed patches with similar morphological or semantic properties into one feature vector per slot.Normalization by routed-weight sums keeps slot-token magnitude stable across different assignment counts.
- Robust regularization for semantic slots: Semantic affinity clustering regularizes routing to prevent slot collapse and preserve representational capacity across the available slots.The auxiliary objective includes load balancing, entropy regularization, and a z-loss.
- Robust regularization for semantic slots: The total training objective combines the downstream task loss with weighted switching, entropy, and z-loss regularizers.The auxiliary terms are jointly optimized with the primary VLM task loss.
3 Experiments and Results
Experiments show that TC-SSA achieves strong slide-level performance under severe token constraints, generalizes across datasets and encoders, and benefits from a controlled semantic-slot budget.
- Efficiency-Performance Comparative Studies: 78.34% overall accuracy is achieved on SlideBench(TCGA), with TC-SSA outperforming competitors in comparative evaluations.The benchmark compares efficiency and performance using Flops and accuracy, including zero-shot datasets.
- Efficiency-Performance Comparative Studies: 78.34% overall accuracy is obtained using only 32 visual tokens, corresponding to a 58× compression ratio versus original patch features.Full-WSI inference is infeasible because of out-of-memory constraints.
- Efficiency-Performance Comparative Studies: 10.64% higher accuracy and 98.3% fewer input tokens are reported against sampling baselines on SlideBench(TCGA).TC-SSA condenses global evidence into K tokens and uses linear-time complexity O(N · K).
- Semantic Slot Analysis: Patches routed to the same slot cluster across datasets in t-SNE space, indicating consistent grouping of similar tissue patterns.The clusters correspond to different tissue semantics.
- Slot-Budget Ablation: K = 32 is the default slot budget; larger budgets provide limited gains and can over-fragment semantic evidence or aggregate background noise.For TCGA-BRCA, performance drops at K = 128, while TCGA-NSCLC gains remain marginal beyond K = 32.
- Encoder Ablation: TC-SSA retains state-of-the-art performance when the CONCH encoder is replaced with UNI, demonstrating encoder-level generalizability.The comparison uses the same UNI encoder as recent baselines.
4 Conclusion
TC-SSA compresses gigapixel pathology representations through semantic slot aggregation and achieves strong performance under strict token budgets. Its fixed slide-level slot budget remains dependent on the patch encoder and may trade fine-grained spatial geometry for semantic structure.
- Method: TC-SSA is a token compression module for gigapixel pathology VLMs that uses semantic slot aggregation.The method is designed for constrained visual-token budgets.
- Results: 78.34% overall accuracy and 77.14% Diagnosis accuracy are achieved on SlideBench(TCGA).The method also reaches AUC of 95.83% on TCGA-BRCA and 98.27% on TCGA-NSCLC.
- Limitations: A fixed slide-level slot budget makes compression quality dependent on the patch encoder and may affect localization-heavy tasks by trading spatial geometry for semantic structure.This is identified as a current limitation of the approach.