PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis

S. M. Nasif Uddin, Rusab Sarmun, Muhammad E. H. Chowdhury, Adam Mushtak, Israa Al-Hashimi, Sohaib Bassam Zoghoul

arXiv:2609.28732v1eess.IVcs.CV

TL;DR

Radiograph-based bone-tumor diagnosis is challenged by heterogeneous lesions and the limits of single-view modeling. The paper combines cropped lesions and full radiographs in a multi-task framework for segmentation and subtype classification. It reports strong segmentation and classification results, including high malignant osteosarcoma AUC.

  • Problem

    Radiograph-based diagnosis is difficult for heterogeneous tumors, and single-input models can lose either small-lesion detail or wider anatomical context.

    Source: §1 · p. 1

  • Method

    The framework pairs ROI crops with full radiographs and uses dual-input multi-task learning to perform segmentation and multi-class classification.

    Source: §1 · p. 2

  • Results

    The model achieved a Dice score of 0.896 and IoU of 0.821, with malignant AUC of 0.999.

    Source: §3.2 · p. 16, §5 · p. 21

  • Takeaways & Limitations

    The authors conclude that contextual integration within radiographs alone can rival or surpass multimodal systems.

    Source: §5 · p. 21

    Future work includes integrating detection, expanding and balancing the dataset across tumor sites, and optimizing the architecture for clinical deployment.

    Source: §4 · p. 21

Abstract

from arXiv · show

Primary bone tumors are rare but clinically aggressive neoplasms whose diagnosis from radiographs is challenged by heterogeneous morphology, subtle lesion margins, and overlapping bone structures. To address the limitations of existing single-view models, we present a dual-input, multi-task learning framework that, to our knowledge, is the first to apply bidirectional cross-modal attention between a lesion crop and the full radiograph for joint segmentation and subtype classification. Using the multi-institutional Bone Tumor X-ray Radiograph Dataset (BTXRD, n=3,746), we employ a YOLO-based detector to generate regions of interest, which are paired with full images as inputs to a dual-stream DenseNet121 architecture. Features are integrated via a novel cross-modal attention fusion strategy, refined by Hierarchical Multi-scale Feature Fusion, effectively balancing fine-grained lesion detail with global anatomical context. Evaluated on a held-out patient-level test split, the model demonstrates superior performance over single-input baselines, achieving an overall Dice Similarity Coefficient of 0.896 and a macro-averaged classification F1-score of 0.928. Notably, the system exhibits exceptional sensitivity for malignant osteosarcoma (AUC 0.999), validating the potential of dual-stream context modeling to support radiologists in accurate, early decision-making.

1. Introduction

Bone-tumor diagnosis from radiographs is difficult because lesions vary widely and can resemble normal or benign structures, while public annotated datasets have been scarce. The paper proposes a dual-input, multi-task framework to combine local lesion detail with global skeletal context for segmentation and multi-class diagnosis.

1. Introduction

  • Public bone-tumor datasets have often been small, restricted, or inaccessible, limiting reproducibility and model generalizability.The cited example, IIEST, contains 69 cases.Source: §1 · p. 2
  • Single-view models trade local detail against anatomical context, while many prior bone-tumor CAD systems focused on benign-versus-malignant classification.Full-image downsampling can obscure small lesions; ROI-only models lose context such as lesion location and periosteal reaction.Source: §1 · p. 2
  • The proposed framework jointly performs tumor segmentation and multi-class classification using cropped ROIs and full radiographs.It fuses parallel streams through cross-attention and hierarchical multi-scale feature fusion before task-specific heads.Source: §1 · p. 2
  • Cross-modal fusion and self-attention are designed to preserve spatial relationships for complex lesions while maintaining semantic consistency for classification.The approach also incorporates hierarchical multi-scale fusion and CBAM attention in encoder, decoder, and fusion blocks.Source: §1 · p. 2
  • The authors report superior diagnostic performance on BTXRD and high sensitivity for clinically relevant subtypes, including osteosarcoma.The supplied introduction does not give numerical results for this comparison.Source: §1 · p. 3

2. Methodology

The methodology proceeds from the BTXRD dataset and class taxonomy through preprocessing and ROI generation to OsteoHiFuse-Net, its multi-task objective, and evaluation metrics.

2. Methodology

  • The methodology is organized around the dataset and three-class taxonomy, preprocessing and patient-level splitting, the OsteoHiFuse-Net architecture, its objective, and evaluation metrics.The pipeline description identifies ROI generation, bidirectional cross-modal attention, hierarchical fusion, segmentation and classification heads, and deep supervision.Source: §2 · p. 3
  • The architecture section covers two DenseNet121 streams, bottleneck fusion, an attention-gated segmentation decoder, and a hierarchical multi-scale classifier fusion module.These components are presented in the sequence used by the pipeline.Source: §2 · p. 3
  • The methodology also specifies a compound multi-task objective and the metrics used to evaluate segmentation and classification.Source: §2 · p. 3

2.1. Dataset Description

BTXRD supplies multi-institutional radiographs with localization, segmentation, and clinical annotations. The study groups diagnoses into three classes and describes the dataset’s anatomical coverage.

2.1.1. BTXRD Overview

  • BTXRD contains 3,746 radiographs from five institutions, including 1,879 normal and 1,867 tumor cases, with tumor cases divided into 1,525 benign and 342 malignant.Images include expert bounding boxes, pixel-level masks, and clinical metadata.Source: §2.1.1 · p. 3
  • The dataset covers multiple anatomical sites, with tibia, femur, fibula, and humerus the most common locations.Images reflect variation in equipment and acquisition parameters across institutions.Source: §2.1.1 · p. 3

2.1.2. Class Distribution and Clinical Relevance

  • The study focuses on osteosarcoma, osteochondroma, and other benign tumors, with the latter category comprising giant cell tumors and simple bone cysts.Osteosarcoma is described as the most common primary malignant tumor; osteochondroma is the most prevalent benign tumor.Source: §2.1.2 · pp. 3–4

2.2. Data Preprocessing Pipeline

The preprocessing pipeline consolidates benign labels, generates detector-based ROIs paired with full radiographs, separates data at the patient level, and applies training-only augmentation and sampling.

2.2.1. Class Consolidation Strategy

  • The final three-class taxonomy groups giant cell tumors and simple bone cysts as other benign, while excluding other severely underrepresented subtypes.The authors state that this grouping introduces intra-class heterogeneity and reflects a clinical decision pathway prioritizing malignant versus non-malignant status.Source: §2.2.1 · p. 4

2.2.2. Automated Tumor Localization and ROI Generation

  • YOLOv11 detects tumors to create standardized ROI crops, which are paired with their corresponding full radiographs for dual-input processing.The paired inputs provide local tumor morphology and margins alongside global anatomical context.Source: §2.2.2 · pp. 4–5
  • The detector was trained only on the training split, with no validation or test images seen during detector training.The authors describe this separation as a measure to prevent data leakage.Source: §2.2.2 · p. 4
  • ROI generation expands detections by 10%, squares the region around the detection centroid, and resamples it to 256 × 256 pixels.The pipeline handles multiple detections independently and manually reviews crops with detector confidence below 0.5.Source: §2.2.2 · pp. 4–5

2.2.3. Train–Validation Split

  • The data split assigns 80% of patients to training and 10% each to validation and testing, keeping all radiographs from each patient in one subset.Augmentation is applied only after splitting and only to training data.Source: §2.2.3 · p. 5

2.2.4. Preprocessing and Augmentation

  • Both cropped ROIs and full radiographs are resized to 256 × 256 pixels, with aspect ratio preserved by padding for full images.Validation and test sets remain unaltered, and augmentations are selected to retain anatomical plausibility and diagnostic features.Source: §2.2.4 · p. 5
  • Training augmentation simulates positioning, exposure, detector noise, and mild blur, while corresponding masks receive identical transformations.Transformations include rotation, horizontal and occasional vertical flipping, brightness and contrast changes, gamma correction, Gaussian noise, and blur.Source: §2.2.4 · p. 5
  • Class-specific sampling after augmentation produced 5,108 training samples and an approximately 1:1:2 ratio across osteochondroma, osteosarcoma, and other benign classes.The text describes this as improving balance while retaining the natural prevalence hierarchy of benign over malignant cases.Source: §2.2.4 · p. 5

2.3. Proposed Dual-Input Multi-Task Architecture

OsteoHiFuse-Net processes cropped tumor regions and full radiographs in separate streams, then fuses them to support segmentation and classification. Its architecture combines cross-modal attention with multi-scale integration and attention-guided decoding.

2.3.1. Overall Framework Design

  • The framework combines dual-stream encoding, bottleneck cross-modal fusion, multi-scale classification features, an attention-gated segmentation decoder, and dual-task outputs.The outputs include a pixel-wise tumor mask, three-class classification, and an auxiliary classification branch.Source: §2.3 · p. 6, §2.3.1 · pp. 6–7

2.3.2.1 DenseNet121 Backbone Selection

  • DenseNet121 is selected for dense connectivity, which supports feature reuse and gradient flow in medical imaging with limited training data.The encoders also use ImageNet-pretrained weights for transfer learning.Source: §2.3.2.1 · p. 7

2.3.2.2 Dual-Stream Configuration

  • DenseNet121 encoders process the crop and full radiograph separately, preserving scale-specific features for subtle tumor patterns and broader anatomical relationships.The crop pathway focuses on high-resolution intra-tumoral patterns, while the full-image pathway captures coarser context.Source: §2.3.2.2 · p. 7, §2.3.2.1 · p. 7

2.3.2.3 Progressive Feature Enhancement

  • Progressive feature enhancement applies depthwise separable convolution, batch normalization, and ReLU before CBAM attention at each encoder stage.The enhanced features are used for skip connections or further downsampling.Source: §2.3.2.3 · p. 7

2.3.2.4 CBAM Attention Integration

  • CBAM recalibrates encoder features through sequential channel and spatial attention to emphasize informative channels and tumor regions.Channel attention uses average and max pooling with a shared MLP; spatial attention uses pooled maps and a 7×7 convolution.Source: §2.3.2.4 · pp. 7–8

2.3.3. Cross-Modal Attention Fusion Module

  • At the bottleneck, cross-modal attention integrates crop and full-image features to model relationships that simple concatenation or addition would miss.The fusion targets the complementary local detail and global context of the two streams.Source: §2.3.1 · p. 6, §2.3.3 · p. 8

2.3.3.1 Query-Key-Value Architecture

  • Bidirectional cross-attention lets crop features query full-image context and full-image features query tumor-region detail.The projected features are reshaped into sequences and processed with eight attention heads, each with d_k = 64.Source: §2.3.3.1 · pp. 8–9

2.3.3.2 Adaptive Feature Fusion

  • The attended feature maps are concatenated and refined into a 512-channel fused bottleneck representation that feeds the decoder.Crop-to-full attention weights are also returned for optional visualization of attended full-image regions.Source: §2.3.3.2 · p. 9

2.3.4. Hierarchical Feature Fusion (HiFuse) Module

HiFuse aggregates crop, cross-modal, and decoder features so classification can use both semantic and fine-grained morphological information. The decoder also applies attention to encoder skip features, while separate heads produce segmentation and classification outputs.

2.3.4.1 Hierarchical Fusion Strategy

  • Each feature map is pooled to 16×16, projected to about 170 channels, spatially attended, concatenated, channel-reweighted, and reduced to a 512-channel descriptor.The final fusion uses a 3×3 convolution and adaptive pooling to 8×8.Source: §2.3.4.1 · p. 10

2.3.4. Hierarchical Feature Fusion (HiFuse) Module

  • HiFuse combines crop bottleneck semantics, cross-modal fused local and global information, and decoder texture and boundary features for classification.The three branches span high-level semantics to fine-grained morphology.Source: §2.3.4 · pp. 9–10, §2.3.4.1 · p. 10

2.3.5.1 Enhanced Attention Gates

  • Attention gates use decoder context to select informative encoder skip regions, and CBAM further refines the attended features.Enhanced convolution blocks then combine these skips with upsampled decoder features.Source: §2.3.5.1 · pp. 10–11

2.3.5.2 Decoder Channel Progression

  • The decoder reduces channels as spatial resolution rises; its final 16-channel feature map retains spatial detail and multi-scale context for pixel-wise segmentation.The supplied channel example reduces the deepest stage from 512 to 112 channels.Source: §2.3.5.2 · p. 11

2.3.6.3 Auxiliary Classification Head

  • The auxiliary classifier reads crop-encoder bottleneck features directly, bypasses fusion, and supplies an additional training gradient and classification regularization.It uses global average pooling and a single linear layer; its prediction is not used at inference.Source: §2.3.6 · p. 11, §2.3.6.3 · p. 12

2.3.6. Multi-Task Output Heads

  • The model branches into segmentation, main classification, and auxiliary classification heads, with the auxiliary output reserved for training.The reported three-class prediction comes from the main head after HiFuse.Source: §2.3.6 · p. 11

2.3.6.1 Segmentation Head

  • The segmentation head converts full-resolution decoder features to pixel logits, applies spatial attention, and thresholds sigmoid probabilities at 0.5 for binary masks.The 7×7 attention map emphasizes coherent tumor regions and suppresses isolated false positives.Source: §2.3.6.1 · p. 11

2.3.6.2 Main Classification Head

  • The main classification head globally pools HiFuse features and uses a three-layer MLP to predict probabilities for osteosarcoma, osteochondroma, and other benign tumors.Softmax is applied during inference.Source: §2.3.6.2 · pp. 11–12

2.3.7. Deep Supervision Strategy

Deep supervision attaches auxiliary segmentation outputs to intermediate decoder stages and combines their losses with the main segmentation objective. The architecture also uses parameter- and computation-conscious design choices intended to support clinical deployment.

2.3.7. Deep Supervision Strategy

  • Auxiliary segmentation heads at decoder stages 4, 3, and 2 are upsampled to input resolution and trained with the main segmentation loss using decreasing weights.Their weighted loss is combined with the main segmentation loss using a 0.4 coefficient.Source: §2.3.7 · p. 12
  • Deep supervision is intended to improve decoder gradient flow and multi-scale feature learning, regularize predictions across scales, and accelerate convergence.The supplied text describes these as expected benefits of intermediate error signals.Source: §2.3.7 · p. 12

2.3.8. Computational E

  • DenseNet121 streams, depthwise separable convolutions, lightweight attention, a single-layer auxiliary head, and a narrowing decoder are used to limit computational cost.CBAM adds spatial and channel attention for under a 1% parameter increase.Source: §2.3.8 · p. 13
  • YOLOv11 localization is described as real-time, so ROI extraction is intended not to bottleneck the diagnostic workflow.The design choices are presented as supporting computational efficiency for clinical deployment.Source: §2.3.8 · p. 13

2.4. Loss Function Design

The training objective balances segmentation and two classification losses, using overlap- and hard-example-sensitive segmentation loss alongside class-balanced focal classification loss. Training also uses weighted sampling, regularization, and a warmup–cosine learning-rate schedule.

2.4.1. Multi-Task Loss Formulation

  • The total training objective combines three loss components weighted α=0.4, β=0.5, and γ=0.1; classification (β=0.5) is weighted more heavily than segmentation (α=0.4).Weights were set based on empirical tuning and task importance in the clinical workflow: accurate classification guides treatment decisions, while segmentation delineates surgical margins and provides spatial information for surgical planning.Source: §2.4.1 · p. 13

2.4.2. Unified Focal Loss for Segmentation

  • Unified Focal Loss combines Dice overlap optimization with focal emphasis on hard examples to address pixel imbalance and small tumor regions.The focal focusing parameter is γ=2.0, and the two terms are balanced with αfocal=1.0.Source: §2.4.2 · p. 13, §2.4.2.1 · p. 13, §2.4.2.2 · p. 14

2.4.2.2 Focal Loss Component

  • The segmentation loss targets severe pixel imbalance and small lesions, which can otherwise bias predictions toward background or lead to under-segmentation.Focal loss down-weights easy examples and focuses learning on hard misclassified pixels such as ambiguous boundaries.Source: §2.4.2 · p. 13, §2.4.2.2 · pp. 13–14

2.4.3. Class-Balanced Focal Loss for Classification

  • Classification uses Class-Balanced Focal Loss to address benign-majority sample imbalance and emphasize difficult subtype cases.Its focusing parameter is γ=3.0, compared with γ=2.0 for segmentation.Source: §2.4.3 · p. 14, §2.4.3.2 · p. 14

2.4.4. Training Configuration and Optimization

  • Classification training uses dropout of 0.5 to limit overfitting, alongside weight decay, batch normalization, gradient clipping, and mixed precision.Training used a maximum of 20 epochs with early stopping based on validation loss.Source: §2.4.3 · p. 14, §2.4.4 · p. 15

2.4.3.1 Class Weight Selection

  • Post-augmentation class weights are 0.8 for osteochondroma, 3.0 for osteosarcoma, and 1.0 for other benign tumors.Osteosarcoma is up-weighted given its clinical importance and 25.0% share of training data.Source: §2.4.3.1 · p. 14

2.4.4. Training Configuration and Optimization

  • A warmup–cosine schedule raises the learning rate linearly for five epochs and then smoothly decays it to the minimum value.The passage describes this schedule as supporting stable convergence.Source: §2.4.4 · p. 15
  • Training used weighted random sampling to balance classes in mini-batches and classification weights of [0.8, 3.0, 1.0] in the order osteochondroma, osteosarcoma, other benign.The reported class shares were 28.4%, 25.0%, and 46.6% in that class order.Source: §2.4.4 · p. 15

2.5. Evaluation Metrics

The evaluation uses overlap metrics for segmentation and class-sensitive measures for classification. Per-image overlap scores are averaged, while classification metrics are computed per class to complement overall accuracy on imbalanced data.

2.5.1. Segmentation Metrics

  • Segmentation evaluation reports Dice Similarity Coefficient and Intersection over Union, with DSC as the primary overlap metric.Both are computed per image and averaged across the validation set using a 0.5 probability threshold.Source: §2.5.1 · p. 15
  • DSC measures predicted-versus-ground-truth foreground overlap on a 0-to-1 scale, with values above 0.9 described as excellent segmentation.IoU is a complementary, stricter overlap measure, especially for small regions.Source: §2.5.1 · p. 15

2.5.2. Classification Metrics

  • Classification evaluation includes accuracy and per-class precision, recall, and F1-score to characterize class-specific performance.Precision measures correctness among predicted class members, recall measures identification among true class members, and F1 balances both.Source: §2.5.2 · pp. 15–16
  • Accuracy alone can mislead on imbalanced data because predicting only the majority class may still yield high accuracy.Per-class measures supplement accuracy in this setting.Source: §2.5.2 · p. 15

3. Results and Discussion

OsteoHiFuse-Net combines local lesion detail with full-radiograph context across detection, segmentation, and classification. The reported results include strong held-out performance, progressive gains from model components, and comparisons with prior studies.

3.1. YOLOv11 ROI Detection Results

  • The YOLOv11x detector reached 75.5% mAP@0.5, 72.5% recall, 76.4% precision, and 41.2% mAP@0.5:0.95 on BTXRD validation.The authors present these results as supporting automated ROI extraction despite the challenges of precise localization.Source: Table 3 · p. 16, §3.1 · p. 16

3.2. Tumor Segmentation Results

  • The segmentation module achieved an overall Dice of 0.896 and IoU of 0.821, with Dice values of 0.923 for osteosarcoma, 0.882 for osteochondroma, and 0.947 for other benign lesions.The authors attribute the higher benign-lesion Dice partly to smoother, better-defined cortical margins; osteosarcoma boundaries are often irregular and permeative.Source: §3.2 · p. 16
  • Qualitative masks showed most errors at lesion boundaries, while predictions remained internally consistent and free from scattered noise.The discussion links contextual information from the full radiograph to distinguishing tumor invasion from structural overlap.Source: §3.2 · p. 16, Figure 7 · p. 17

3.3. Tumor Classification Results

  • Classification achieved a macro-averaged F1-score of 0.928 and macro AUC of 0.993; one-vs-rest AUC values were 0.993, 0.999, and 0.987, respectively.Osteosarcoma and osteochondroma were correctly classified in 0.917 and 0.966 of instances, respectively, and other benign lesions in 0.931.Source: §3.3 · pp. 16–17, Figure 8 · p. 18
  • Osteosarcoma recall was 0.917, while other benign lesions had recall of 0.931 and precision of 0.818; some aggressive tumors with low-contrast edges were interpreted as benign-like.The authors describe overlap between osteosarcoma and benign cases, particularly for borderline lesions with partial cortical involvement or irregular calcification.Source: §3.3 · p. 18

3.4. Ablation Study

  • Adding the crop branch improved Dice by 1.9 percentage points and F1-score by 5.7 percentage points; the full model with HiFuse achieved an overall F1-score of 0.928.The authors report further metric gains with CBAM and describe HiFuse as producing the most balanced performance across metrics.Source: §3.4 · pp. 18–19, Table 6 · p. 19

3.5. Comparison with Existing Literature

  • OsteoHiFuse-Net achieved 0.955 multiclass accuracy and 0.993 macro AUC, and 0.955 accuracy and 0.999 AUC for malignant-versus-rest classification using radiographs only.The authors report that its performance can match or exceed more complex multimodal systems.Source: Table 7 · p. 20, §3.5 · p. 20

4. Limitations and Future Work

The framework has limitations in detection, dataset balance, computational efficiency, and its radiograph-only scope. Future work targets integrated detection, broader site-balanced data, architectural optimization, and complementary imaging.

4. Limitations and Future Work

  • Small tumors with low contrast or subtle cortical disruption are sometimes missed because detection is a preprocessing step, not an integrated learning objective.The authors propose incorporating detection into a unified architecture to share features and reduce cascading errors.Source: §4 · p. 20
  • Limited examples of each tumor subtype at each anatomical location may impair generalization to unseen tumor types or rare anatomical regions.The imbalance constrains learning location-specific radiographic variation, including cortical structure and background bone density.Source: §4 · p. 20
  • Processing two parallel feature streams increases memory and computational cost, potentially limiting real-time deployment in low-resource settings.Suggested optimizations include model compression, attention pruning, and lightweight fusion.Source: §4 · p. 21
  • Radiograph-only analysis cannot capture some soft-tissue or intramedullary features visible in MRI or CT.The authors suggest exploring hybrid frameworks that add complementary modalities when available.Source: §4 · p. 21
  • Future work targets integrated detection, more balanced tumor-site data, and model-efficiency optimization for real-world workflows.The authors also propose location-aware evaluation and selective use of complementary modalities.Source: §4 · p. 21

5. Conclusion

OsteoHiFuse-Net combines lesion detail and skeletal context for bone-tumor analysis from radiographs. It reports strong segmentation and classification results, while detection performance and dataset balance remain limitations.

5. Conclusion

  • Dice score 0.896, IoU 0.821, overall accuracy 0.955, and malignant AUC 0.999 were achieved by OsteoHiFuse-Net.The authors state that contextual integration within radiographs alone can rival or surpass multimodal systems.Source: §5 · p. 21
  • OsteoHiFuse-Net integrates local lesion detail with global skeletal context for bone-tumor analysis on radiographs.The model produced anatomically consistent boundaries in benign and malignant lesions, including morphologically complex cases.Source: §5 · p. 21
  • Detection performance and dataset balance remain limitations, motivating work on detection integration, site diversity, and model efficiency.The authors identify real-time clinical use as a focus for efficiency optimization.Source: §5 · p. 21

Data Availability

The study uses the publicly available BTXRD dataset and collected no new patient data. The class subset, consolidation rules, and patient-level split are described in Sections 2.1 and 2.2.

Data Availability

Loading 2609.28732v1…