PaperlayerView exact revision on arXiv

Source-linked AI summary

CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification

Bingxin Yu, Xueli Wang, Jerry Zhou, Wenyan Wang, Li Wen, Lan Huang, Xin Feng, Fengfeng Zhou, Kewei Li

arXiv:2608.13939v1cs.CVcs.AI

TL;DR

Most thyroid ultrasound models emphasize binary malignancy classification, leaving five-class TI-RADS prediction and feature-level supervision underexplored. CMCNet aligns image embeddings with textual TI-RADS representations, with auxiliary supervision yielding consistent, robust improvements across encoders.

  • Problem

    Five-class TI-RADS prediction and structured feature-level supervision remain underexplored, while available datasets lack complete annotations for all five TI-RADS levels.

  • Method

    CMCNet aligns image embeddings with fixed textual embeddings of standardized TI-RADS descriptions using a Center-Margin Contrastive Loss.

  • Results

    Auxiliary loss provided the most consistent and robust improvements across encoders.

  • Takeaways & Limitations

    The STN dataset supplies structured feature-level supervision for thyroid ultrasound representation learning while requiring only images at inference.

  • Takeaways & Limitations

    The ablation indicates that CMCNet effectiveness depends on auxiliary supervision, with simpler appearance attributes less effective without it.

Abstract

from arXiv · show

Ultrasound is the primary imaging modality for assessing thyroid nodules, and the ACR TI-RADS framework standardizes diagnosis through five ultrasound feature categories that are aggregated into five risk levels (TR1-TR5). Although widely adopted in clinical practice, most deep learning approaches focus on binary malignancy classification, while multi-class prediction and explicit utilization of feature-level supervision remain underexplored, largely due to limited annotated data. In this study, we introduce the STN dataset of 600 thyroid nodules with paired transverse and longitudinal ultrasound images, bounding box annotations, and complete labels for all five TI-RADS feature categories. Following the clinical decision process, we investigate how structured feature information can guide representation learning during training while requiring only images at inference. We demonstrate that text embeddings derived from standardized feature descriptions form a stable surrogate representation for TI-RADS risk levels. Based on this observation, we propose CMCNet, which aligns image embeddings to fixed textual embeddings via a Center-Margin Contrastive Loss that simultaneously promotes intra-class compactness and inter-class separation. Experimental results show that this embedding alignment strategy is more data-efficient and robust than direct multitask learning, and consistently outperforms InfoNCE, center loss, a strong multitask baseline, and a VQA-style multimodal model, particularly in imbalanced settings. The dataset is freely available at doi: 10.5281/zenodo.19125693 and the source code is available at: https://www.healthinformaticslab.org/supp/.

1 Introduction

The paper addresses limited TI-RADS-consistent multi-class modeling by introducing the 600-nodule STN dataset and CMCNet, which transfers structured textual supervision to image embeddings for image-only inference. CMCNet aligns images with fixed textual anchors using CMCLoss and outperforms alternative alignment, multitask, and multimodal approaches, especially under imbalance.

  • Problem: Existing thyroid ultrasound models mainly optimize binary decision-making, while few perform full five-class TI-RADS prediction or explicitly model all five feature categories.Prior work using TI-RADS descriptors focused on a reduced subset of risk levels rather than full five-class prediction.
  • Dataset: The STN dataset contains 600 nodules with paired transverse and longitudinal images, bounding boxes, complete annotations for all five TI-RADS feature categories, and corresponding risk levels.It enables systematic investigation of structured, TI-RADS-consistent five-class modeling.
  • Textual supervision: Standardized textual descriptions encoded with frozen ModernBERT achieve near-perfect TI-RADS classification accuracy, indicating a compact surrogate space for risk discrimination.The text representations are used to transfer structured semantic information into image representations during training because inference uses images alone.
  • Method: CMCNet aligns image embeddings with fixed text embeddings through CMCLoss, promoting intra-class compactness, inter-class separation, and attraction to textual anchors.An additional L1 regularization term stabilizes gradient dynamics because inter-text distances are smaller than image-embedding distances; inference remains image-only.
  • Results: CMCLoss provides more stable, discriminative alignment than InfoNCE and center loss, while CMCNet consistently outperforms FLaMM-Net and LLaVA-Med, particularly under imbalanced conditions.These findings support fixed semantic anchors as an effective inductive bias for small medical datasets governed by structured decision rules.

2 Related Work

Prior TI-RADS studies largely use binary classification, while this work introduces a publicly reported dataset with complete feature-level annotations and explicitly aligns ultrasound images with structured text. The section reviews relevant biomedical and ultrasound encoders, text embeddings, and representation-learning losses.

  • Most existing TI-RADS prediction studies formulate the task as binary classification.
  • This work presents the first publicly reported thyroid ultrasound dataset annotated with scores for all five TI-RADS feature categories and overall risk levels.
  • It is also the first work described here to explicitly align ultrasound images with structured textual representations for TI-RADS classification.
  • Image encoders: BiomedCLIP aligns biomedical images and text in a shared embedding space, but its pretraining does not include ultrasound images.
  • Image encoders: The work provides the first systematic evaluation of recent foundation and transformer-based models for TI-RADS multi-class classification on thyroid ultrasound images.
  • Text encoder: ModernBERT generates frozen embeddings for standardized TI-RADS descriptions, with text embeddings alone achieving approximately 98% accuracy on validation and test sets.These results indicate that structured text provides a highly separable surrogate space for TI-RADS risk levels.

3 Materials and Methods

The Materials and Methods section introduces the retrospectively collected STN dataset with paired ultrasound views, nodule localization, and complete TI-RADS annotations, then defines CMCNet’s multimodal alignment and comparison architectures. It also describes hierarchical multitask variants and motivates the Center-Margin Contrastive Loss for aligning image embeddings with textual representations.

  • Dataset construction: Cases required complete transverse and longitudinal views of one dominant nodule, with the most suspicious nodule selected for patients having multiple nodules.The selection followed routine diagnostic practice.
  • Dataset construction: The STN dataset includes paired transverse and longitudinal ultrasound images, manually annotated nodule bounding boxes, and labels for five TI-RADS feature categories.The categories are composition, echogenicity, shape, margin, and echogenic foci.
  • Annotation and labeling: TI-RADS feature scores are assigned according to ACR TI-RADS criteria and linearly summed to determine the final TR1–TR5 level.At least two experienced ultrasound physicians independently assessed annotations and risk levels, with consensus readings for disagreements.
  • CMCNet architecture: CMCNet encodes both ultrasound views with a shared image encoder, converts structured TI-RADS scores into text descriptions using a frozen text encoder, and aligns both modalities in a shared embedding space.The alignment uses the proposed Center-Margin Contrastive Loss.
  • Model variants: Multi-head version 1 predicts all tasks from a shared latent representation, whereas version 2 learns feature-specific representations before concatenating them to predict the final TI-RADS category.Version 2 follows the clinical scoring workflow more closely.

4 Experimental Results · 4.1 Text Modality Provides a Near-Upper-Bound Representation · 4.2 Direct Optimization of the Target from Images

Experiments show that structured TI-RADS text embeddings provide a near-upper-bound, nearly linearly separable target representation, while direct image optimization benefits most from cropping, multi-view input with auxiliary supervision, and stronger multimodal constraints.

  • 4.1 Text Modality Provides a Near-Upper-Bound Representation: Text embeddings form compact, well-separated clusters for the five TI-RADS levels, with minimal inter-class overlap and near-perfect diagonal confusion-matrix dominance.The representation is described as compact and linearly separable, with only marginal errors between adjacent risk levels.
  • 4.2.1 Image-Only Baselines and Ablation Design: The image-only FLaMM-Net study evaluates single or paired ultrasound views, optional nodule bounding-box crops, and shared or feature-specific multi-head designs with focal-loss auxiliary supervision.The configurations reflect the structured TI-RADS scoring rule and address imbalance across the five feature categories.
  • 4.2.2 Which Image Encoder Performs Best?: 75.17% balanced accuracy, 73.80% F1, and 91.78% weighted F1 make USFM the strongest overall encoder, narrowly ahead of BiomedCLIP’s 74.81%, 73.15%, and 88.70%.BiomedCLIP performs best on shape and margin, whereas USFM leads composition and echogenicity; SAMUS leads TI-RADS but degrades on several attribute heads.
  • 4.2.3 Module-Wise Ablations of FLaMM-Net: Bounding-box cropping improves most tasks for both encoders, indicating that restricting inputs to the nodule region reduces background noise and stabilizes feature learning.The improvement is particularly evident when cropping is combined with auxiliary supervision.
  • 4.2.3 Module-Wise Ablations of FLaMM-Net: Multi-view input has mixed effects without auxiliary supervision but becomes consistently beneficial across encoders and head variants once auxiliary loss is introduced.Its gains are most apparent for margin and TI-RADS, while composition and echogenicity may degrade without auxiliary supervision.
  • 4.2.3 Module-Wise Ablations of FLaMM-Net: Auxiliary loss provides the most stable improvements, whereas consistency loss is configuration-dependent and can suppress complementary information between views.Direct feature-head optimization can still trade off TI-RADS separation, motivating stronger intermediate constraints through multimodal alignment.
  • 4.2.4 Summary of the Ablation Study: Across configurations, cropping yields stable gains, multi-view effectiveness depends on auxiliary supervision, and auxiliary loss is more robust than consistency loss.These patterns summarize the limitation of direct image-only optimization and motivate multimodal alignment with stronger intermediate constraints.

4.3 Evaluation of Intermediate Embedding Alignment · 4.4 CMCLoss as an Effective Image–Text Alignment Objective · 4.5 Embedding Alignment vs. Direct Image-Only Optimization

Embedding alignment stabilizes intermediate representation learning, making auxiliary and consistency supervision more effective and reducing encoder-dependent degradation. CMCLoss further improves structured image–text alignment, while CMCNet generally outperforms direct image-only optimization and LLaVA-Med.

  • 4.3 Evaluation of Intermediate Embedding Alignment: CMCNet aligns image embeddings with structured textual embeddings, constraining intermediate representations and reducing representation uncertainty.This framework re-evaluates auxiliary supervision, consistency regularization, and multi-head design under text alignment.
  • 4.3 Evaluation of Intermediate Embedding Alignment: Auxiliary supervision produces more consistent gains in CMCNet than FLaMM-Net, with nearly all BiomedCLIP configurations improving TI-RADS prediction and all improving feature-level tasks.Similar trends are observed for USFM, indicating that text-derived semantic centers stabilize feature-level optimization.
  • 4.3 Evaluation of Intermediate Embedding Alignment: Under embedding alignment, consistency loss has predominantly positive or neutral effects, with particularly clear improvements for margin and echogenic foci.Constraining embeddings toward shared textual anchors reduces intra-class variance and supports more reliable alignment.
  • 4.3 Evaluation of Intermediate Embedding Alignment: The preferred multi-head structure depends on the encoder: BiomedCLIP favors Multi-head v1, whereas USFM benefits more from Multi-head v2.Under alignment, Multi-head v2 no longer exhibits FLaMM-Net’s degradation and consistently outperforms its image-only counterpart.
  • 4.4 CMCLoss as an Effective Image–Text Alignment Objective: CMCLoss, which jointly enforces center aggregation and margin separation, achieves the strongest overall performance among InfoNCE, Center Loss, and its L1 and L2 variants.For BiomedCLIP with Multi-head v1, L2 performs best overall; for USFM with Multi-head v2, L1 yields the best average and beats the other objectives on five of six tasks.
  • 4.4 CMCLoss as an Effective Image–Text Alignment Objective: CMCLoss produces tighter cross-modal aggregation and reduced intra-class dispersion, whereas InfoNCE can preserve a modality gap by emphasizing negative-pair separation.L2_CMCL yields the lowest intra-class dispersion for both encoders, and smaller distances to corresponding text centers consistently accompany stronger downstream performance.
  • 4.5 Embedding Alignment vs. Direct Image-Only Optimization: CMCNet with aligned embeddings improves over FLaMM-Net in most task–metric combinations, outperforming it on five of six tasks across all three metrics in each matched encoder setting.BiomedCLIP uses L2-CMCL with Multi-head v1, while USFM uses L1-CMCL with Multi-head v2; margin and composition are the main exceptions, respectively.
  • 4.5 Embedding Alignment vs. Direct Image-Only Optimization: LLaVA-Med remains below FLaMM-Net and CMCNet across metrics and encoders, supporting explicit embedding alignment as a more direct inductive bias than generative VQA.The passage attributes this comparison to limitations involving natural-image-pretrained vision encoders, small imbalanced datasets, shortcut learning, and language priors.

5 Conclusion

The study introduces the STN dataset and CMCNet, showing that structured textual embedding alignment improves image-only TI-RADS classification and clinical interpretability. Future work will expand validation and investigate stronger alignment and pretraining methods.

  • Dataset and motivation: STN provides 600 single-nodule cases with transverse and longitudinal ultrasound views, bounding boxes, and expert-labeled TI-RADS attributes.The dataset addresses the lack of structured feature-level supervision in thyroid ultrasound datasets.
  • Structured supervision: Textual representations of five ultrasound feature categories accurately predict TI-RADS categories, indicating that structured descriptions encode sufficient diagnostic information.Text embeddings are used during training rather than inference, because direct reliance on them is not clinically meaningful at deployment.
  • Embedding alignment: CMCNet’s embedding alignment consistently outperforms direct multitask optimization across attribute-level and TI-RADS tasks, especially for margin and echogenic foci.Alignment to structured textual anchors constrains embeddings toward the TI-RADS scoring framework and improves clinical interpretability.
  • Optimization objective: CMCLoss jointly enforces intra-class compactness and cross-modal alignment, outperforming InfoNCE and Center Loss across configurations.The loss provides a practical objective for structured image-text alignment.
  • Future work: Future work will expand STN to larger multicenter and multinodule cohorts, pursue external validation, and investigate improved alignment objectives and thyroid-ultrasound pretraining.The framework may ultimately support a dedicated TI-RADS foundation model and a structured, clinically oriented diagnostic question-answer system.

Declaration of Statements · Ethics Statement

The study received institutional review board approval, and written informed consent was waived because the research was retrospective and used fully anonymized patient data.

  • Ethics Statement: The study was approved by the Institutional Review Board of China-Japan Union Hospital of Jilin University.The approval number was 2026030504.
  • Declaration of Statements: Institutional approval was granted by China-Japan Union Hospital of Jilin University.The reported approval number is 2026030504.
  • Ethics Statement: The ethics review covered a retrospective study design.The retrospective design contributed to the consent waiver.
  • Ethics Statement: The study used fully anonymized patient data.Data anonymization was part of the basis for waiving written informed consent.
  • Ethics Statement: The consent waiver applied because the study was retrospective and used fully anonymized patient data.Both conditions are explicitly stated as the rationale for waiving written consent.
  • Ethics Statement: The requirement for written informed consent was waived.The waiver was associated with the study’s retrospective design and use of fully anonymized patient data.

8 Appendix

The appendix reports the experimental setup, including hardware, stratified data splitting, warmup, and model-specific learning-rate choices. FLaMM-Net and CMCNet used identical splits for fair comparison, while USFM and BiomedCLIP required different learning-rate treatments.

  • FLaMM-Net and CMCNet used identical data splits to ensure a fair comparison.
  • The STN dataset was stratified by TI-RADS category and split into training, validation, and test sets at a 10:1:1 ratio.
  • Linear learning-rate warmup lasted 5% of the total training iterations.
  • USFM used a relatively small learning rate to preserve pretrained representations and maintain training stability, whereas BiomedCLIP was fine-tuned with a larger learning rate.

Appendix Table A1. Hyperparameter settings for FLaMM-Net and CMCNet.

Model selection uses validation TI-RADS classification accuracy, while final results are reported on a held-out test set. During inference, CMCNet relies only on ultrasound images; feature annotations and point-based scores are training-only.

  • Model selection and evaluation: The checkpoint with the highest validation TI-RADS classification accuracy was selected for final evaluation.All reported results were obtained from the held-out test set.
  • Inference requirements: During inference, the model uses only ultrasound images, without requiring feature annotations or point-based scoring information.The five key ultrasound feature annotations and point-based scoring information are used only during training.
  • Feature annotations: STN ultrasound feature categories support supervised training and evaluation, whereas location serves as contextual metadata outside TI-RADS scoring.The categorical codes correspond to the five key ultrasound attributes in the TI-RADS scoring framework.

Appendix Table A2. Ultrasound feature coding scheme.

PCA visualization shows that frozen ModernBERT text embeddings form compact, well-concentrated clusters, whereas CMCNet image embeddings are more dispersed. This contrast highlights embedding alignment as a means of reducing the modality gap.

  • Embedding visualization: PCA projects CMCNet image embeddings from BiomedCLIP and USFM, together with pre-projection text embeddings from frozen ModernBERT, into two dimensions.The projection is used for qualitative inspection.
  • Embedding structure: Text embeddings form compact, well-concentrated clusters that reflect a structured, low-variance semantic representation.They are described as tightly clustered in the visualization.
  • Embedding structure: Image embeddings exhibit a broader spatial distribution because of variability from visual appearance and acquisition factors.The image embeddings are more dispersed than the text embeddings.
  • Embedding alignment: The contrast between tightly clustered text embeddings and dispersed image embeddings highlights embedding alignment in reducing the modality gap.This interpretation follows directly from the PCA visualization description.
Loading 2608.13939v1…