PaperlayerView exact revision on arXiv

Source-linked AI summary

From Image Latent Space to Fuzzy Rules: Interpretable Analysis of Gastrointestinal Foundation Model

Michael D. Vasilakakis, Dimitris K. Iakovidis

arXiv:2610.00414v1cs.CV

TL;DR

Understanding how pretrained foundation models organize clinically relevant information in their features remains challenging in safety-critical medical imaging. This paper reads frozen patch features through clustered prototypes and linguistic fuzzy rules, then evaluates the approach across GI tasks and uses the rules to inspect synthetic images. The rule-based readout is reported as comparable in accuracy to black-box classifiers, while domain-specific pretraining yields discriminative, symbolically compressible features.

  • Problem

    How pretrained models organize clinically relevant structures in their features, and how that organization changes across depth, remains difficult to understand in safety-critical domains.

    Source: §1 · p. 2

  • Method

    The framework clusters class-specific prototypes from frozen foundation-model patch features and classifies prototype similarities with sparse linguistic IF-THEN rules.

    Source: §1 · p. 2

  • Results

    Across GI classification and polyp segmentation, the rule-based readout is comparable in accuracy to black-box classifiers, while domain-specific pretraining tends to yield discriminative, symbolically compressible latent spaces.

    Source: §1 · pp. 2–3, §5 · p. 14

  • Takeaways & Limitations

    Rules fitted on real data can also give a human-readable, rule-level account of which prototypes and rules synthetic images fail to reproduce.

    Source: §1 · p. 3, §5 · p. 14

    For classification tasks, image-level labels are broadcast to patches, so normal and background patches in abnormal images are also labeled abnormal; segmentation instead uses pixel-mask-derived patch labels.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Foundation models pretrained on large-scale datasets demonstrate strong transferability to medical imaging tasks. However, understanding how their latent representations encode clinically relevant information remains an open challenge in safety-critical domains. This study proposes a prototype-based fuzzy-rule framework that interprets the patch-level features produced by the inner layers of pretrained foundation models, without any fine-tuning. Class-specific prototypes are learned by clustering in the feature space, yielding compact visual patterns. Patch features are then expressed as prototype similarities and classified by fuzzy rules with linguistic IF-THEN conditions that are human readable. The framework is applied across the final two blocks of ViT-S/16 backbones pretrained on ImageNet-1K and GastroNet-5M, and benchmarked against k-nearest neighbours, kernel SVM, and linear probing under identical frozen features, on wireless capsule endoscopy classification, gastrointestinal endoscopy classification, and colonic polyp segmentation. The experimental analysis shows that the proposed method, without backbone fine-tuning, reaches accuracy comparable to these black-box classifiers, and that domain-specific pretraining yields features that are both discriminative and symbolically compressible. Because the resulting rules are extracted from real data and expressed in interpretable terms, they are further used as an instrument to investigate synthetic medical images, providing a human-readable account of which real prototypes and rules a generator reproduces or fails to reproduce, localising where a synthetic image departs from real tissue rather than summarising it with a single score. The framework thus offers a transparent, depth-resolved view of how foundation models organise clinically relevant structure, together with a practical downstream use of the extracted rules.

1 Introduction

The study addresses the difficulty of interpreting clinically relevant structure in pretrained foundation-model features and how it changes with depth. It introduces a frozen-feature prototype and fuzzy-rule analysis, reports comparable classification performance, and applies the rules to synthetic-image analysis.

1 Introduction

  • Interpreting latent representations remains challenging in safety-critical GI imaging, while saliency methods can miss representation-space structure and may reflect biases rather than causal relationships.The introduction motivates structured prototype and fuzzy-rule explanations as alternatives to black-box decisions.Source: §1 · pp. 2–3
  • The framework clusters class-specific prototypes from frozen patch features, converts features to prototype similarities, and uses sparse linguistic IF-THEN rules for human inspection.It examines the final two ViT-S/16 blocks under ImageNet-1K and GastroNet-5M pretraining against classifiers evaluated on identical features.Source: §1 · p. 2
  • Across three GI tasks, the rule-based readout achieves accuracy comparable to black-box classifiers while producing compact, human-readable rules.The analysis covers wireless capsule endoscopy and GI endoscopy classification and colonic polyp segmentation, without backbone fine-tuning.Source: §1 · p. 2
  • Domain-specific pretraining tends to yield latent spaces that are both discriminative and symbolically compressible, with fewer and shorter rules describing the same structure.This pattern is reported across depths and pretraining sources.Source: §1 · p. 3
  • The extracted rules are reused to audit synthetic images, naming real prototypes and rules that generators over-activate or fail to activate, and identifying fidelity–utility gaps.This provides a human-readable account of departures from real tissue beyond aggregate generative metrics.Source: §1 · p. 3

2 Related Work

Prior work offers prototype-based explanations, frozen-feature interpretability, fuzzy rules, and synthetic-image evaluation, but these lines differ in how they treat backbone adaptation, representation depth, and interpretability. The paper situates its approach as a linguistic, layer-resolved analysis with per-prototype and per-rule synthetic-data comparisons.

Prototype-based interpretable classification

  • Unlike end-to-end prototype networks, this study uses frozen features to separate representation analysis from backbone adaptation and examines multiple depths and pretraining domains.Several prototypes per class address visually diverse classes; comparisons to end-to-end prototype networks are avoided because they conflate feature quality with backbone adaptation.Source: §2 · p. 3

Post-hoc prototype interpretability

  • Compared with post-hoc prototype work on trained models, the framework describes similarity with linguistic fuzzy sets rather than heatmaps and compares trained layers systematically.The cited prior approaches include and.Source: §2 · p. 4

Fuzzy rule-based interpretability

  • The fuzzy rules adapt a structured-data rule-construction framework [12] to frozen ViT patch features, using graded linguistic membership to retain partial matches.The graded view also supports tolerant descriptions of synthetic data.Source: §2 · p. 4

Evaluating synthetic medical images

  • Unlike synthetic-image metrics that summarize quality with a single score, the rule base provides human-readable, per-prototype and per-rule accounts of real–synthetic divergence.The cited evaluation approaches include distributional distances, embedding-based precision and recall, real-versus-synthetic discrimination, and train-on-synthetic/test-on-real protocols.Source: §2 · p. 4

3 Methodology

The method turns frozen patch features into class-specific prototypes, linguistic similarity features, and compact fuzzy rules. Those rules provide patch-level class decisions that can be aggregated for image classification or lesion masks.

3.1 Latent Feature Extraction and Prototype Construction

  • K-medoids clusters class-labeled patch features from frozen ViT-S/16 into prototypes; classification labels are weakly broadcast, while segmentation labels come from pixel masks.The weak-label assumption can mark normal and background patches in an abnormal image as abnormal.Source: §3.1 · pp. 4–5

3.2 Prototype-Based Latent Representation

  • Prototype similarities are normalized across the vocabulary and to [0,1], then clustered per prototype into ordered linguistic terms with fuzzy membership functions.The terms provide human-readable descriptions such as Low and High.Source: §3.2 · pp. 5–6

3.3 Interpretable Classification

  • Class evidence weights prototype proximity by fuzzy-label membership; signed pairwise certainty drives predictions, while deduplicated training patterns form compact IF-THEN rules.The extracted rule base is assembled from training patches and held fixed for describing unseen real images and auditing synthetic ones.Source: §3.3 · pp. 6–7

4 Experiments and results

Across three GI benchmarks, frozen-feature fuzzy rules are competitive with black-box classifiers; the experiments also examine rule compactness and use real-data rules to audit synthetic images. Domain-specific pretraining tends to yield features that are both discriminative and symbolically compressible.

4.1 Experimental setup

  • Across KID, Kvasir-v2, and Kvasir-SEG, evaluation uses frozen ViT-S/16 features from two depths and pretraining sources, comparing fuzzy rules with k-NN, kernel SVM, and linear probing.Classification uses image-level label broadcast to patches; segmentation uses mask-derived patch labels, with a stratified 40% test split and ten repeats.Source: §4.1 · pp. 7–8

4.2 Classification and Segmentation Performance

  • On pooled abnormal-versus-normal classification, GN-5M fuzzy rules stay within a few AUC points of trained probes on KID and near 0.99 on Kvasir-v2, while matching or exceeding k-NN.Under IN-1K pretraining, the gap to probes widens, but fuzzy rules remain ahead of k-NN; both depths show essentially the same pattern.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9
  • For image-level classification, L=3 is best at almost every prototype count and keeps rules shorter, whereas segmentation benefits from larger L.P=7 is the sole classification exception to the L=3 pattern.Source: §4.2 · pp. 10–11

4.3 Interpretability Analysis

  • The extracted real KID rule base has 35 rules, each with at most six conditions, and scoring it reproduces the reported classifier decisions exactly.Each prototype is represented by its nearest real training-patch crop, grounding linguistic conditions in tissue examples.Source: §4.3 · p. 11, Figure 3 · p. 11

4.4 Synthetic-image audit via fuzzy rule fidelity

  • The unchanged real-data rules audit synthetic images by comparing per-patch rule-firing and prototype-activation frequencies with real references, naming specific divergences.The audit averages frequencies across sampled images; Fig. 4 connects a single StyleGAN patch to a matching real prototype.Source: §4.4 · p. 12, Table 6 · p. 12, Table 7 · p. 12
  • For KID polyps, StyleGAN’s mean abnormal-patch vote is 85 versus 124 for real polyps, and prototype p6 activation falls from 10% to 0%.For Kvasir-v2 ulcers, the vote is 141 for StyleGAN versus 132 for real, with dominant rules and prototypes matching.Source: §4.4 · p. 13

5 Conclusion

The framework turns frozen foundation-model patch features into compact, human-inspectable fuzzy rules, with accuracy comparable to black-box classifiers across GI tasks. The same rules can also inspect synthetic images for specific departures from real tissue.

5 Conclusion

  • Clustering real patch features yields linguistic prototype similarities and short fuzzy rules, each condition grounded in a concrete real-tissue crop.Rule length remains within the stated 7±2 comprehension budget.Source: §5 · p. 14
  • Across three GI tasks, the rule readout was comparable to black-box classifiers, trailing by a few AUC points on classification and about two Dice points on domain-backbone segmentation.It also outperformed k-nearest neighbours; the penultimate block matched or slightly exceeded the final block, and domain-specific pretraining yielded discriminative, symbolically compressible representations.Source: §5 · p. 14
  • The real-data rule base can read synthetic images and name the prototypes and rules that a generator fails to reproduce.Worked examples showed both a localized failure and close agreement.Source: §5 · p. 14
  • Future work will evaluate rule explanations with clinicians and extend the synthetic-image audit beyond illustrative single-image cases to more images and generators.The proposed broader study would make the rule-level reading a systematic fidelity instrument.Source: §5 · p. 14

Related papers

Loading 2610.00414v1…