PaperlayerView exact revision on arXiv

Source-linked AI summary

HoVer-Trans: Anatomy-aware HoVer-Transformer for ROI-free Breast Cancer Diagnosis in Ultrasound Images

Yuhao Mo, Chu Han, Yu Liu, Min Liu, Zhenwei Shi, Jiatai Lin, Bingchao Zhao, Chunwang Huang, Bingjiang Qiu, Yanfen Cui, Lei Wu, Xipeng Pan, Zeyan Xu, Xiaomei Huang, Zaiyi Liu, Ying Wang, Changhong Liang

arXiv:2205.08390v2eess.IVcs.CV

TL;DR

Breast ultrasound diagnosis faces reader variability, limited specialist availability, black-box models, and limited open data. HoVer-Trans encodes anatomical priors in an ROI-free model and is reported to perform comparably across datasets while outperforming two senior sonographers under the same condition.

  • Problem

    BUS assessment relies on sonographers, with specialist scarcity, intra- and interobserver variability, black-box models, and limited open datasets identified as concerns.

    Source: §I · p. 1

  • Method

    HoVer-Trans is an ROI-free, anatomy-aware model that uses horizontal and vertical strip embeddings to represent inter-layer and intra-layer spatial relationships.

    Source: §III · p. 3, §VI · p. 10

  • Results

    HoVer-Trans achieved comparable quantitative performance across datasets and outperformed two senior sonographers on the full dataset and BI-RADS subgroup analysis under the same condition.

    Source: §I · p. 2

  • Takeaways & Limitations

    GDPH&SYSUCC adds an open breast lesion dataset, and HoVer-Trans provides interpretable evidence to support model decisions.

    Source: §I · p. 2

Abstract

from arXiv · show

Ultrasonography is an important routine examination for breast cancer diagnosis, due to its non-invasive, radiation-free and low-cost properties. However, the diagnostic accuracy of breast cancer is still limited due to its inherent limitations. It would be a tremendous success if we can precisely diagnose breast cancer by breast ultrasound images (BUS). Many learning-based computer-aided diagnostic methods have been proposed to achieve breast cancer diagnosis/lesion classification. However, most of them require a pre-define ROI and then classify the lesion inside the ROI. Conventional classification backbones, such as VGG16 and ResNet50, can achieve promising classification results with no ROI requirement. But these models lack interpretability, thus restricting their use in clinical practice. In this study, we propose a novel ROI-free model for breast cancer diagnosis in ultrasound images with interpretable feature representations. We leverage the anatomical prior knowledge that malignant and benign tumors have different spatial relationships between different tissue layers, and propose a HoVer-Transformer to formulate this prior knowledge. The proposed HoVer-Trans block extracts the inter- and intra-layer spatial information horizontally and vertically. We conduct and release an open dataset GDPH&SYSUCC for breast cancer diagnosis in BUS. The proposed model is evaluated in three datasets by comparing with four CNN-based models and two vision transformer models via five-fold cross validation. It achieves state-of-the-art classification performance with the best model interpretability. In the meanwhile, our proposed model outperforms two senior sonographers on the breast cancer diagnosis when only one BUS image is given.

I. INTRODUCTION

Breast ultrasound is useful for diagnosis, but evaluation faces reader variability, limited specialist availability, black-box models, and limited open data. The paper introduces an ROI-free, anatomy-aware model and a new dataset, reporting competitive performance and interpretable evidence.

I. INTRODUCTION

  • BUS is low-cost, non-invasive, radiation-free, and relatively sensitive for dense breast tissue, motivating precise diagnosis from ultrasound images.The paper highlights the relevance of this diagnostic goal for dense-breast patients in Asia.Source: §I · p. 1
  • BUS assessment depends on sonographers, while specialist scarcity, reader variability, limited open datasets, and black-box models remain concerns.The authors state that clinical usability should be further assessed.Source: §I · p. 1
  • The authors release GDPH&SYSUCC, a two-center dataset of 2,405 BUS images, comprising 1,519 malignant and 886 benign images without ROI annotations.The paper describes it as the largest open breast lesion classification dataset in the field.Source: §I · pp. 1–2
  • HoVer-Trans encodes breast anatomical priors to extract inter-layer spatial information horizontally and intra-layer information vertically.The paper describes malignant tumors as invading deeper layers and benign tumors as disrupting gland continuity.Source: §I · p. 2
  • HoVer-Trans outperformed two senior sonographers on the full dataset and BI-RADS subgroup analysis under the same condition.The authors also report comparable quantitative performance across datasets and attention to malignant lesion boundaries.Source: §I · p. 2

II. RELATED WORKS

  • Earlier BUS classification work includes transfer learning, ensembles, image enhancement, multimodal prediction, region-focused methods, and dual-backbone approaches.The cited examples include a large private dataset, illustrating that open data availability remains a concern.Source: A. Breast Cancer Diagnosis in… · p. 2

B. Transformer-based Medical Image Classification

The paper frames anatomy-aware modeling as a way to combine interpretable features with predictive ability. Its methodology represents breast anatomy through horizontal and vertical strip embeddings alongside patch embeddings.

B. Transformer-based Medical Image Classification

  • Vision transformers apply self-attention to image tiles, while medical-image transformer models have been developed for tasks including CT, pathology, and fundus classification.The cited examples include COVID-VIT, TransMIL, and MIL-VT.Source: B. Transformer-based Medical Image… · p. 3
  • The authors seek interpretable features to support decisions while considering predictive ability, generalizability, and convenience.They contrast this goal with designing a complex black-box breast cancer prediction model.Source: B. Transformer-based Medical Image… · p. 3

III. METHODOLOGY

  • The methodology leverages visible breast anatomy as prior knowledge for a fully automatic ultrasound diagnosis model.The paper proceeds from anatomy-aware formulation to the HoVer-Trans stage and overall network structure.Source: §III · p. 3
  • Unlike ViT's 16×16 patches, the proposed anatomical representation uses horizontal strips for inter-layer and vertical strips for intra-layer visual words.Source: Figure 3 · p. 4

A. Anatomy-aware Formulation

Breast ultrasound anatomy provides a prior: tissue layers and lesion relationships matter for malignancy, while conventional CNNs represent spatial relationships less effectively. The section frames HoVer-Trans as a ViT-based approach that uses horizontal and vertical strips to encode those relationships.

A. Anatomy-aware Formulation

B. HoVer-Trans Stage

1) Vision Transformer:

  • ViT divides an image into patches, flattens them into visual-word tokens, and uses self-attention to build spatial correlations across tokens.Source: 1) Vision Transformer · p. 4
  • HoVer-Trans uses ViT blocks with the same structure but omits the class embedding.Source: 1) Vision Transformer · p. 4

2) Embedding:

The embedding scheme represents BUS images as patches and as horizontal and vertical strips. HoVer-Trans combines these representations through four branches that extract and fuse spatial features across and within anatomical layers.

2) Embedding:

  • Patch embedding divides the BUS image into patches indexed by row and column, then flattens and linearly projects them into 1D vectors.Source: 2) Embedding · p. 4
  • Horizontal strip embedding represents visual words from the same anatomical layer using M strips.Source: 2) Embedding · p. 4
  • Vertical strip embedding represents visual words across anatomical layers using M strips.Source: 2) Embedding · p. 4

3) HoVer-Trans Block:

  • A HoVer-Trans block has four symmetric branches: horizontal (H), vertical (V), horizontal-to-vertical (H2V), and vertical-to-horizontal (V2H).Source: 3) HoVer-Trans Block · p. 4
  • At the first block of a stage, the four branch inputs equal the features from the preceding stage.Source: 3) HoVer-Trans Block · p. 4
  • The H and V auxiliary branches extract inter-layer and intra-layer spatial correlations using horizontal and vertical strip embeddings.Source: 3) HoVer-Trans Block · pp. 4–5
  • The anatomy-aware features from the auxiliary branches feed the main branches and also serve as inputs to the next HoVer-Trans block.Source: 3) HoVer-Trans Block · p. 5
  • H2V and V2H are the main feature-extraction branches, fusing features from the H and V auxiliary branches.Source: 3) HoVer-Trans Block · p. 5

4) Conv Block:

A convolutional block follows the last HoVer-Trans block at each stage to fuse branch features and add inductive bias. It processes reshaped, concatenated feature maps through three convolutional layers.

4) Conv Block:

  • The convolutional block follows the final HoVer-Trans block in each stage to fuse H2V and V2H features and introduce inductive bias.Source: 4) Conv Block · p. 5
  • The block uses three convolutional layers: a 1 × 1 layer doubles channels, a 3 × 3 layer, and a final 1 × 1 layer compresses channels for the next stage.Source: 4) Conv Block · p. 5
  • The two main-branch feature vectors are reshaped into 2D maps, concatenated, and passed to the convolutional block.Source: 4) Conv Block · p. 5

C. Overall Architecture

The model stacks four stage modules, each containing HoVer-Trans blocks, a convolutional block, and pooling. Feature-map resolution is progressively reduced, and the final stage uses a fully connected layer for inference.

C. Overall Architecture

  • The model comprises four stage modules, each with several HoVer-Trans blocks, one convolutional block, and one pooling layer.Source: C. Overall Architecture · p. 5
  • A convolutional stem downsamples the BUS image to H/4 × W/4 × C, followed by stage resolutions H/8 × W/8 × 2C, H/16 × W/16 × 4C, and H/32 × W/32 × 8C.Source: C. Overall Architecture · p. 5
  • Convolutional blocks connect adjacent stages to fuse horizontal and vertical information; the last stage uses a fully connected layer for inference and cross-entropy loss for optimization.Source: C. Overall Architecture · p. 5

D. Implementation Details

The section records model-training settings and dataset context, including UDIAT and the scope of the multi-dataset comparison table.

D. Implementation Details

  • HoVer-Trans used four stages with embedding dimensions {4, 8, 16, 32}, block counts {2, 4, 4, 2}, and head counts {2, 4, 8, 16}.Training used AdamW for 250 epochs, batch size 32, weight decay 0.1, 10 warm-up epochs, and an initial learning rate of 0.0001 with cosine decay; images were resized to 256×256.Source: D. Implementation Details · p. 5

IV. DATASETS

  • The study evaluates its model on three datasets, including two public datasets and one constructed dataset.UDIAT is described as the first public dataset.Source: §IV · p. 5, A. UDIAT · p. 5

A. UDIAT

  • UDIAT contains 163 BUS images: 109 benign and 54 malignant.The images came from the UDIAT Diagnostic Centre in Sabadell, Spain.Source: A. UDIAT · pp. 5–6
  • Table II compares state-of-the-art approaches across three datasets, reports DeLong-test p-values for AUC comparisons, and marks models requiring predefined ROIs.The caption identifies ROI-based models with † and significance thresholds with asterisks.Source: Table II · p. 6

B. BUSI

The section describes BUSI and the GDPH&SYSUCC dataset, then outlines preprocessing and experimental evaluation, including the metrics and visualizations used.

B. BUSI

  • BUSI contributes 647 lesion images to the study after excluding 133 normal images from its original 780-image collection.The analyzed set contains 210 benign and 437 malignant images; each image has pixel-level lesion ground truth.Source: B. BUSI · p. 6

C. GDPH&SYSUCC

  • GDPH&SYSUCC contains 2,405 pathology-labeled images from two medical centers: 1,519 malignant and 886 benign.The malignant images came from 676 patients and the benign images from 526 patients.Source: C. GDPH&SYSUCC · p. 6

1) Data Preprocessing:

  • Images were converted from DICOM to PNG, UI regions were excluded by rectangle detection, and foreground completeness was manually checked.Source: 1) Data Preprocessing · p. 6

V. EXPERIMENTAL RESULTS

  • Experiments compare quantitative results and heatmap visualizations across three datasets, include a sonographer comparison and ablations, and show heatmaps throughout to assess interpretability.Source: §V · p. 6, Figure 5 · p. 7

1) Evaluation Metrics:

2) Competitors:

HoVer-Trans is compared with ROI-based and ROI-free approaches, then assessed on classification and heatmap interpretation, including against sonographers on GDPH&SYSUCC.

2) Competitors:

  • The comparison includes ResNet50, VGG16, ViT, TNT-s, JBHI2020, and MICCAI2020; the latter two require predefined lesion ROIs.In GDPH&SYSUCC, the study also tested JBHI2020 using sonographer-labeled bounding boxes.Source: 2) Competitors · p. 7

1) Performance of deep-learning models:

  • 0.855 ACC, 0.876 precision, 0.867 recall, and 0.872 F1-score were HoVer-Trans's best reported BUSI values.Its BUSI performance was comparable to ROI-based JBHI2020.Source: 1) Performance of deep-learning models · p. 7
  • On GDPH&SYSUCC, HoVer-Trans achieved the best AUC, accuracy, precision, recall, and F1-score among the baseline models, including JBHI2020.The paper notes that ROI-free analysis retains spatial information around tumors, which may be useful when training data are sufficient.Source: 1) Performance of deep-learning models · pp. 7–8
  • HoVer-Trans heatmaps showed more accurate lesion locations and focused attention than the compared models; the paper links its horizontal and vertical formulation to analysis of tissue-layer relationships.The figure overlays feature-map heatmaps on BUS images, with lesions indicated by white arrows.Source: 1) Performance of deep-learning models · p. 8, Figure 6 · p. 8

C. HoVer-Trans vs. Sonographers

  • HoVer-Trans achieved more precise diagnostic performance than two experienced sonographers on the GDPH&SYSUCC dataset.The comparison aggregated five-fold model predictions; Table III covers the full dataset and two BI-RADS subgroups.Source: C. HoVer-Trans vs. Sonographers · p. 8, Table III · p. 8
  • The reader comparison was limited to one static image per assessment, unlike sonographers’ usual dynamic examination across multiple views.Source: C. HoVer-Trans vs. Sonographers · p. 8
  • Reader consistency was low in BI-RADS 2&3, while HoVer-Trans had more stable performance across both BI-RADS subgroups under the single-image condition.The reported examples include specificity of 0.996 for reader 1 versus 0.879 for reader 2, and recall of 0.099 versus 0.455.Source: C. HoVer-Trans vs. Sonographers · p. 8

D. Ablation Studies

The ablations test the anatomy-aware formulation, the convolutional connection between stages, and transformer configurations.

1) Effectiveness of Anatomical Prior Knowledge:

2) Effectiveness of Conv Block:

  • Removing the convolutional layer decreased quantitative performance and caused attention maps to lose focus.The study compared the convolutional block with average pooling and a 1×1 convolution.Source: 2) Effectiveness of Conv Block · p. 9
  • The complete HoVer-Trans interaction of horizontal and vertical anatomy-aware formulation achieved the best classification results and most interpretable attention maps.Source: 2) Effectiveness of Conv Block · p. 9

3) Sizes of Different Embedding Ways:

The embedding-size experiments select p = 2 and h&v = 2 for the final model. The conclusion describes HoVer-Trans’s anatomy-aware design, future multi-view plans, and weaker performance on smaller datasets.

3) Sizes of Different Embedding Ways:

  • p = 2 and h&v = 2 achieve the best classification performance among tested embedding configurations and are used in the final model.The authors select h&v = 2 to avoid strips that are too narrow or computationally costly, and p = 2 to match the strip-token size.Source: 3) Sizes of Different Embedding Ways · p. 9

VI. CONCLUSION

  • HoVer-Trans combines transformer and CNN components, using patch, horizontal-strip, and vertical-strip embeddings to model inter- and intra-layer spatial correlations.The model is ROI-free and provides interpretable attention maps to support predictions.Source: §VI · p. 10
  • The authors propose aggregating BUS images from multiple views and tracking probe imaging signals for comprehensive diagnosis, if hardware permits.These are proposed future improvements to the single-image assessment.Source: §VI · p. 10
  • The model shows poor classification performance when trained on smaller datasets such as UDIAT.The authors plan to construct a larger multi-center dataset to further explore capacity and generalizability.Source: §VI · p. 10
Loading 2205.08390v2…