PaperlayerView exact revision on arXiv

Source-linked AI summary

TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation

Hai Pham Ngoc

arXiv:2609.28590v1cs.LGcs.CV

TL;DR

Thyroid cytology AI must limit false negatives and handle overconfidence under clinical domain shift. TAM-Chain combines multi-scale Markov stopping with entropy-based uncertainty and specialist referral. It reports zero internal false negatives and adaptive referral and observation behavior under external validation.

  • Problem

    Thyroid cytology AI faces false-negative risks and performance degradation with overconfident predictions under external domain shift.

    Source: §1.4 · p. 4, §2.3 · p. 5

  • Method

    TAM-Chain uses an entropy-guided Absorbing Markov Chain across magnifications to stop diagnosis automatically or refer uncertain cases to specialists.

    Source: §1.4 · p. 4, §7.1 · p. 16

  • Results

    Macro F1 was 0.9741 with FNR = 0.00% internally, while external validation showed expected steps increasing from 2.0181 to 2.4944 and referrals from 44.65% to 66.84%.

    Source: §1.4 · p. 4, §7.1 · p. 16

  • Takeaways & Limitations

    The reported results support dynamic stopping and specialist referral as safeguards for internal automated decisions and external domain-shift cases.

    Source: §6.1 · p. 13, §7.1 · p. 16

    Under severe domain shift, human referrals rose to 66.84%, reducing AI autonomy; the authors identify domain adaptation as future work.

    Source: §6.4 · p. 16

Abstract

from arXiv · show

Background & Problem: Thyroid Fine-Needle Aspiration Biopsy (FNAB) cytology based on the Bethesda System plays a pivotal role in early thyroid cancer detection; however, deep learning approaches face substantial challenges regarding high false-negative rates and overconfidence under clinical domain shift. Methods: In this study, we propose TAM-Chain, a multi-scale (10x, 20x, 40x) thyroid cytology classification framework leveraging Absorbing Markov Chain theory combined with Shannon Entropy-based Uncertainty Quantification. The framework dynamically models multi-magnification feature extraction as an absorbing stochastic process, enabling optimal stopping criteria and a human-in-the-loop referral mechanism to strictly suppress critical diagnostic errors. Results: Extensive evaluation on an internal test set (N = 235) demonstrates a Macro F1 score of 0.9741 with an absolute False-Negative Rate (FNR) of 0.00%. On an independent external validation set (N = 1015) presenting severe domain shift, TAM-Chain maintains superior stability and classification performance (Macro F1 = 0.7026) by adaptively adjusting the expected stopping step and triggering specialist referrals, significantly outperforming single-magnification baselines. Conclusion: The TAM-Chain framework proves to be a highly effective, safe, and adaptable solution for digital pathology workflows, successfully harmonizing automated diagnostic efficiency with stringent biological safety.

1 Introduction

Thyroid cytology AI must avoid malignant-to-benign errors while balancing multi-scale image context, computational cost, and domain-shift risk. TAM-Chain uses entropy-guided Markov stopping and specialist referral, reporting strong internal results and adaptive external handling.

1.1 Clinical Background and Challenges in Thyroid Cytology Diagnosis

  • False negatives—classifying Bethesda B5/B6 malignancies as benign B2—can defer critical surgical intervention, making reliable referral mechanisms central to clinical safety.The introduction identifies FNR near 0% as a prerequisite for clinical deployment.Source: §1.1 · p. 2

1.2 Limitations of Single-Scale Deep Learning Models and Domain Shift

  • Single-magnification models trade broad architectural context at 10× against nuclear detail at 40×, while high magnification adds computational and memory burdens and can lose cellular context.Domain shift can also degrade performance and produce overconfident diagnostic errors.Source: §1.2 · p. 2

1.3 Proposed Framework: TAM-Chain

  • TAM-Chain combines Shannon Entropy-based uncertainty quantification with Absorbing Markov Chains to stop dynamically across magnifications or refer unresolved cases to expert pathologists.When uncertainty falls below the safety threshold, the chain ends in AI diagnosis; persistent uncertainty after 40× triggers referral.Source: §1.3 · p. 2

1.4 Core Contributions

  • Macro F1 was 0.9741 with FNR = 0.00% on the internal test set (N = 235), while external validation (N = 1015) increased expected steps from 2.0181 to 2.4944 and referrals from 44.65% to 66.84%.The zero FNR applies within the automated decision subset (A1).Source: §1.4 · p. 4
  • The paper also reports 1,000 bootstrap iterations and Friedman tests (p < 0.001), with variance σ2 = 0.000110 for AvgProbs compared with single-scale baselines.These findings are included among the study’s stated contributions.Source: §1.4 · p. 4

2 Related Work

Prior work motivates combining multi-scale cytology, uncertainty estimation, and Markovian decisions. The proposed framework addresses trade-offs in single-scale analysis and overconfidence under domain shift through adaptive stopping, referral, and retraining-free adaptability.

2.1 Thyroid Cytology Diagnosis and Limitations of Single-Scale Deep Learning

  • Single-scale cytology models preserve either broad architecture at 10× or nuclear detail at 40×, with the latter also risking context loss and greater computational memory overhead.The paper identifies multi-scale diagnosis as a clinical-practice challenge.Source: §2.1 · pp. 4–5

2.2 Uncertainty Quantification and Human-in-the-Loop Safeguards

  • Shannon Entropy can quantify uncertainty, and predictions above a safety threshold can be deferred to pathologists through human-in-the-loop referral.The cited related work describes this strategy as suppressing FNR.Source: §2.2 · p. 5

2.3 Domain Shift and the Challenge of Model Overconfidence

  • External domain shift can degrade medical AI performance, while overconfident networks may assign erroneous predictions confidence above 95%, defeating maximum-softmax confidence gating.The cited sources describe cross-institutional variations in scanners, preparation, section thickness, and staining as sources of feature shifts.Source: §2.3 · p. 5

2.4 Absorbing Markov Chains in Multi-Tiered Decision Optimization

  • Coupling Absorbing Markov Chains with Shannon Entropy as state-transition triggers across 10×, 20×, and 40× magnifications is presented as a novel research paradigm.The related-work discussion notes Markovian models in treatment planning and disease-progression forecasting.Source: §2.4 · p. 5

2.5 Comparative Synthesis and Research Gap

  • TAM-Chain’s stated research-gap response unifies dynamic stochastic early stopping, automated expert referrals, and adaptability under domain shift without model retraining.Table 1 is identified as a systematic comparison of existing methodologies and TAM-Chain.Source: §2.5 · pp. 5–6, Table 1 · p. 6

3 Methodology

TAM-Chain processes aligned FNAB patches at increasing magnifications, extracting probabilities with a specialized backbone and using entropy to guide uncertainty-aware decisions. The design pairs broad-to-fine image context with computational deployment choices.

3.1 Architectural Overview of the TAM-Chain Framework

  • The framework combines deep-learning morphological feature extraction with Absorbing Markov Chains for dynamic early stopping and safe referral across 10×, 20×, and 40× tiers.At each tier, the network outputs a predicted probability distribution.Source: §3.1 · p. 6

3.2 Backbone Architecture and Model Deployment (ThyroidEffi)

  • ThyroidEffi 1.0 Base is the specialized backbone, built on EfficientNet optimizations with Squeeze-and-Excitation blocks to focus on thyroid-cell nuclear regions.Training uses weighted cross-entropy for Bethesda groups B2, B5, and B6, AdamW, an initial learning rate of 10^-4, and cosine annealing.Source: §3.2 · p. 6, §3.2.1 · p. 7

3.2.1 Backbone Architecture and Training

  • Weighted cross-entropy addresses class imbalance across B2, B5, and B6 during backbone training.Optimization uses AdamW with an initial learning rate of 10^-4 and cosine annealing decay.Source: §3.2.1 · p. 7

3.2.2 Real-Time Deployment via TorchScript

  • TorchScript deployment removes Python interpreter overhead and rapidly extracts tier-specific probability distributions before Shannon Entropy evaluation.Inference follows the 10× → 20× → 40× transition sequence.Source: §3.2.2 · p. 7

3.3 Multi-Scale Patch Extraction and Preprocessing

  • WSI preprocessing partitions slides into non-overlapping or controlled-overlapping patches and constructs image pyramids to retain contextual information and nuclear detail during cropping and resizing.The protocol standardizes inputs across the three magnification tiers.Source: §3.3 · p. 7, §3.3.1 · p. 7

3.3.1 Specimen Normalization and Region of Interest (ROI) Segmentation

  • The 10×, 20×, and 40× tiers progress from wide-field cluster density and architecture, through intermediate cell-cluster context, to fine nuclear features relevant to B5/B6 classification.The fine features listed include psammoma bodies, nuclear grooves, membrane irregularities, and pale chromatin.Source: §3.3.1 · p. 7

3.3.2 Multi-Scale Feature Alignment

  • Patches with identical spatial centers are extracted across magnifications and passed through the backbone to produce probability vectors for entropy computation and Markov state control.Source: §3.3.2 · p. 7

3.4 Uncertainty Quantification via Shannon Entropy

  • Shannon Entropy measures uncertainty at each tier, with low entropy indicating confidence and entropy approaching log2(3) ≈ 1.585 bits indicating class ambiguity.Entropy is compared with a validation-optimized safety threshold to govern chain transitions.Source: §3.4 · pp. 7–8

3.5 Absorbing Markov Chain Formulation

The Markov formulation represents magnification inspections as transient states and AI diagnosis or expert referral as absorbing outcomes. Validation-derived transition probabilities support expected-step and outcome-probability calculations that describe efficiency and safety trade-offs.

3.5.1 State Space Definition

  • At each tier, cases with entropy below the safety threshold enter A1 for automatic diagnosis; unresolved uncertainty at 40× routes cases to A2 for expert referral.The transient states are S1, S2, and S3, corresponding to 10×, 20×, and 40× inspection.Source: §3.5.1 · p. 8, §3.5.2 · p. 9

3.5.2 Transition Probability Matrix

  • The transition matrix P is expressed in canonical form, with Q for transitions among transient states, R for transitions into absorbing states, and zero and identity blocks for absorbing-state rows.Source: §3.5.2 · p. 8
  • Validation data determine the transition probabilities: r_t,A1 is the proportion meeting the safety criterion at tier t, while q_t is the proportion escalating to the next tier.At S3, cases that remain above the entropy threshold are obligatorily referred to A2.Source: §3.5.2 · pp. 8–9

3.5.3 Fundamental Matrix Analysis and Expected Stopping Step 𝔼[𝜏]

  • The Fundamental Matrix N gives expected visits to transient states before absorption, and expected observation steps from S1 are calculated by summing its first row.Source: §3.5.3 · p. 9
  • The absorption matrix B gives the probability that a chain starting in each transient state ends in each absorbing state, including AI decision A1 or referral A2.Source: §3.5.3 · p. 9
  • Lower expected steps indicate computational savings from stopping at lower magnifications, while A1 and A2 probabilities represent AI autonomy and referral-based safety.Source: §3.5.3 · p. 9

3.6 Multi-Scale Probability Aggregation Strategy (AvgProbs)

AvgProbs averages predicted probabilities across all three magnification tiers to provide recommendations for cases referred for expert evaluation. The strategy is described as stabilizing predictions under domain shift.

3.6 Multi-Scale Probability Aggregation Strategy (AvgProbs)

  • AvgProbs computes the unweighted arithmetic mean of predicted probability matrices across all three magnification tiers for referred cases.The averaged probabilities support an AI recommendation intended to minimize context loss during supplementary expert pathologist evaluation.Source: §3.6 · p. 10
  • The final recommendation for set A2 is the class with the highest averaged probability.Source: §3.6 · p. 10
  • AvgProbs is reported to smooth localized 40× artifacts while incorporating 10× spatial context, with minimal prediction variance and greater stability under domain shift.Source: §3.6 · p. 10

4 Experimental Setup

The experiments use internal and independent external FNAB cohorts, calibrate an entropy threshold under a zero-FNR constraint, and compare single-scale and multi-scale diagnostic strategies.

4.1 Datasets and Cohort Partitioning

  • The internal cohort contains 1,804 patients partitioned at the patient level, and the external cohort comprises 1,015 patients for evaluation under severe domain shift.The validation set has N=236 and the internal test set has N=235.Source: §4.1 · p. 10

4.2 Calibration Algorithm for the Safety Threshold 𝜃safe

  • The safety threshold is calibrated on the independent validation set to balance strict FNR suppression against the proportion of cases safely classified without referral.FNR counts ground-truth malignant B5/B6 cases incorrectly predicted as benign B2.Source: §4.2 · p. 10

4.2.1 Search Space and Objective Function

  • The grid search tests thresholds from 0.0000 to 0.1000 in increments of 0.0001, maximizing AI autonomy subject to zero tolerance for false negatives.Source: §4.2.1 · p. 11

4.2.2 Algorithmic Execution Protocol

  • The execution protocol simulates the 10×→20×→40× chain, stopping in A1 when entropy falls below the threshold or referring unresolved cases to A2.Among thresholds satisfying FNR=0.00%, it selects the value maximizing the automated-decision proportion; calibration yielded θsafe*=0.0010 on N=236.Source: §4.2.2 · p. 11

4.3 Baseline Diagnostic Strategies

  • The comparison includes fixed single-scale strategies at 10×, 20×, and 40×, MinEntropy, and multi-scale AvgProbs.Source: §4.3 · p. 11

5 Results

The supplied material identifies Table 2 as the internal-test performance summary but provides no metric values.

5.1 Performance Evaluation on the Internal Test Set (𝑁= 235)

  • Table 2 summarizes comprehensive performance metrics for the internal test set of N=235.Source: Table 2 · p. 11, §5.1 · p. 11
  • The table labels diagnostic strategy and magnification as an evaluation dimension.Source: §5.1 · p. 12
  • Pathologist referral rate is identified as a reported table measure.Source: §5.1 · p. 12

TAM-Chain (AvgProbs)

The external-validation and bootstrap tables cover performance on the N=1015 cohort and variance analysis based on 1,000 resampling iterations; the supplied passages provide no table cell values.

5.2 Performance Evaluation on the External Validation Set (𝑁= 1015)

5.3 Bootstrap Variance Analysis and Statistical Testing

External Val

The Friedman test found significant differences across diagnostic strategies on both evaluation cohorts. The experimental visualizations include normalized confusion matrices and other views of model behavior.

External Val

  • χ2 = 3451.4876 (p < 0.001) on the Internal Test Set and χ2 = 3123.9520 (p < 0.001) on the External Validation Set indicate significant differences across diagnostic strategies.Source: External Val · p. 13

5.4 Experimental Visualizations

  • The experimental visualizations also include Markov absorbing-state flows, stopping-step distributions, and bootstrap variance boxplots.Source: §5.4 · p. 13
  • Figure 2 presents normalized confusion matrices (%) across the internal and external evaluation cohorts.Source: Figure 2 · p. 13, §5.4 · p. 13

6 Discussion

TAM-Chain combines uncertainty-triggered specialist referrals with multi-scale stopping, reporting zero false negatives internally and adjusted observation and referral behavior under external domain shift. AvgProbs shows stronger external performance and stability than MinEntropy, while the evaluation scope and reduced autonomy under severe shift remain limitations.

6.1 Clinical Safety Mechanisms and Internal Performance Analysis

  • FNR = 0.00% and Macro F1 = 0.9741 on the Internal Test Set demonstrate the reported internal safety and classification performance.Source: §6.1 · p. 13
  • When uncertainty reaches H(p_t) ≥ 0.0010, TAM-Chain refers cases to specialists; 44.65% of internal cases were referred, while 47.7% exited at 10×.The reported average expected stopping step was E[τ] = 2.0181.Source: §6.1 · p. 15

6.2 Deciphering Stochastic Domain Shift Adaptation via Markovian Flows

  • Under external domain shift, over 71% of cases shifted to 40×, expected observation steps increased from 2.02 to 2.49, and specialist referrals rose from 44.65% to 66.84%.The Single 40x baseline declined from 0.9065 internally to 0.6420 externally, a 26.4% absolute decrease.Source: §6.2 · p. 15

6.3 Superiority of the AvgProbs Strategy via Bootstrap Variance Analysis

  • Externally, AvgProbs achieved mean F1 = 0.7022 (0.7026 overall) versus MinEntropy's 0.6879, with variance σ² = 0.000179 and SD = 0.0134.Both strategies achieved internal F1 = 0.9741; AvgProbs's external 95% confidence interval was 0.6758–0.7273.Source: §6.3 · p. 16

6.4 Limitations and Potential Remedies

  • Evaluations cover Bethesda B2, B5, and B6; indeterminate categories such as Bethesda III and IV have high morphological ambiguity and would require an expanded Markov state space.Source: §6.4 · p. 16
  • Under severe domain shift, referrals rose to 66.84%, reducing AI autonomy; the authors identify domain adaptation as future work.Source: §6.4 · p. 16

7 Conclusion and Future Work

TAM-Chain combines absorbing Markov chains with Shannon-entropy uncertainty quantification. The conclusion reports zero internal false negatives and adaptive external referrals, while future work proposes broader learning, data fusion, and prospective validation.

7.1 Conclusion

  • FNR = 0.00% within the automated AI decision cohort (A1) on internal testing, with no malignant B5/B6 tumors misclassified as benign.The evaluations covered 1,804 internal patients and 1,015 external patients.Source: §7.1 · p. 16
  • Under external staining and scanner variation, expected observation steps rose from E[tau] = 2.02 to 2.49 and specialist referral rates P(A2) rose from 44.65% to 66.84%.The conclusion describes these adjustments as a safeguard against out-of-distribution noise.Source: §7.1 · p. 16
  • Dynamic early stopping reduced image-processing time versus fixed high-magnification workflows; 1,000 bootstrap iterations found AvgProbs variance of sigma^2 = 0.000110.Friedman testing confirmed statistically significant performance gains (p < 0.001).Source: §7.1 · p. 16

7.2 Future Work

  • Future work proposes learnable Markov transitions and extending the feature backbone and absorbing-state matrices to all six Bethesda categories.The transition approach would move beyond fixed entropy thresholds toward reinforcement learning or attention mechanisms.Source: §7.2 · p. 17
  • Proposed extensions also include combining cytology with ultrasound and clinical metadata, then prospectively validating TAM-Chain in partner-hospital microscopy workflows.The stated aim of multimodal fusion is to enhance diagnostic precision for borderline indeterminate cases.Source: §7.2 · p. 17
Loading 2609.28590v1…