PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Musa Tur Farazi, K G Subarno Bithi

arXiv:2608.21482v1eess.IVcs.AIcs.CV

TL;DR

Multimodal fracture classifiers can become brittle when metadata are missing, corrupted, or anatomically mismatched. This study evaluates reliability- and anatomy-consistency-aware fusion on Bangladeshi radiographs, finding reproducible gains from structured context and reduced degradation under mismatched metadata, at some clean-accuracy cost.

  • Problem

    Metadata can provide useful context for fracture classification but may be unreliable when missing, corrupted, or inconsistent with image anatomy.

  • Method

    The study combines leakage-safe metadata with ConvNeXt image features using reliability-gated residual and hierarchical fusion, plus anatomy-consistency gating and controlled metadata-corruption tests.

  • Results

    Multimodal fusion improved class-balanced and calibration results over image-only learning, while anatomy-consistency gating reduced degradation under mismatched metadata at some clean-accuracy cost.

  • Takeaways & Limitations

    Robustness-aware multimodal learning offers a promising approach for fracture classification when structured metadata may be unavailable or incorrect.

  • Takeaways & Limitations

    The study lacks external validation, so performance across hospitals, devices, adult populations, and other countries remains unknown.

Abstract

from arXiv · show

Background: Multimodal fracture classifiers may benefit from patient and anatomical metadata, but they can also become brittle when contextual information is missing or mismatched. Methods: We studied 1493 radiographs from the Bangladeshi OrthoFrac-XR dataset using leakage-safe age, sex, bone type, and laterality. A ConvNeXt image encoder was combined with a clinical multilayer perceptron through concatenation, late fusion, reliability-gated residual fusion, and a hierarchical state-location formulation. We additionally introduced an anatomy-consistency gate that attenuates metadata corrections when an image-side anatomical prediction disagrees with the reported bone type. Results: Across five folds and three seeds, hierarchical residual fusion achieved a macro-F1 of 0.6046 +/- 0.0279, compared with 0.5727 +/- 0.0270 for image-only learning, while improving the Brier score from 0.5239 to 0.4948. In a five-fold robustness experiment, anatomy-consistency fusion reduced the macro-F1 loss under shuffled metadata from 0.0567 to 0.0203 relative to ordinary residual fusion, although its clean-data macro-F1 was lower. Without bone type at inference, auxiliary anatomy supervision improved macro-F1 from 0.5620 +/- 0.0330 to 0.5899 +/- 0.0289. Conclusions: Structured context improves fracture classification, and consistency-aware gating limits harm from mismatched metadata. The observed clean-performance-robustness trade-off and the absence of patient-level identifiers motivate external and prospective validation.

1 Introduction

The introduction motivates robust multimodal fracture classification because structured context may improve radiographic interpretation but can also create failures when metadata are missing, corrupted, or mismatched. The study therefore combines leakage-safe multimodal benchmarking with reliability-gated, hierarchical, and anatomy-consistency methods focused on robustness as well as accuracy.

  • Motivation: Routine-care context such as age, anatomical site, and laterality may complement image-only fracture models by altering disease prevalence and interpretation.
  • Motivation: Metadata can become a safety liability when models learn contextual shortcuts or receive syntactically complete but incorrect patient or bone-type information.
  • Study contributions: The study benchmarks leakage-safe structured metadata on Bangladeshi radiographs and evaluates reliability-gated residual fusion, hierarchical state-location modeling, and anatomy-consistency extensions.
  • Study contributions: The benchmark uses 1493 radiographs with five-fold and multi-seed evaluation, probability calibration, metadata-only ablations, and subgroup analysis.
  • Study contributions: Multimodal learning provides a reproducible gain over image-only learning, while anatomy-consistency reduces degradation under mismatched metadata without improving clean discrimination over the strongest simple fusion baseline.

2 Materials and Methods … 2.6 Baselines

This retrospective study used leakage-safe metadata, grouped cross-validation, ConvNeXt and clinical encoders, and a set of multimodal and metadata-only baselines to classify four fracture-related classes. The design also explicitly addressed duplicate images, unavailable patient identifiers, and structured-variable leakage.

  • 2.2 Dataset and Cohort: 1493 radiographs from four Bangladeshi hospitals were classified into distal fracture, proximal fracture, post-fracture, and non-fracture.The public release linked radiographs with structured metadata but did not expose patient identifiers.
  • 2.2 Dataset and Cohort: Exact duplicate images were grouped using SHA-1 hashes, detecting four duplicate groups containing eight images, but patient-level grouping remained unavailable.The study explicitly treated the absence of patient identifiers as a limitation.
  • 2.3 Leakage-Safe Clinical Inputs: Only age, sex, bone type, and laterality were used as structured inputs; radiograph-derived measurements, observations, and fracture labels were excluded to prevent leakage.Excluded variables included bone width, fracture gap, gap visibility, primary observation, and the fracture label.
  • 2.4 Data Partitioning: Five stratified, exact-image-grouped folds used approximately 895 training, 299 validation, and 299 test images per split, with three random seeds producing 15 fold–seed evaluations per model.The anatomy-consistency extension used all five folds with seed 2026.
  • 2.5 Image and Clinical Encoders: Images were resized to 224 × 224 pixels, augmented with crops, ±7◦ rotations, and autocontrast, normalized with ImageNet statistics, and encoded by ImageNet-pretrained ConvNeXt-Tiny.Structured inputs were processed by a two-layer MLP with layer normalization, GELU activations, and dropout; both representations entered a shared 256-dimensional fusion space.
  • 2.6 Baselines: The study compared image-only ConvNeXt, a clinical MLP, concatenation, validation-selected late fusion, reliability-gated residual fusion, and hierarchical image-only and residual variants.Concatenation used image, clinical, and availability representations, while late fusion combined image-only and clinical-only probabilities.
  • 2.6 Baselines: Hierarchical image-only and hierarchical residual fusion decomposed the classification task into hierarchical formulations described later.These were included as task-decomposed model variants.
  • 2.6 Baselines: Logistic regression and histogram gradient boosting provided classical metadata-ablation baselines for estimating target information available from structured context alone.These models complemented the clinical MLP in the metadata-only comparison.

2.7 Reliability-Gated Residual Fusion · 2.8 Hierarchical State–Location Prediction · 2.9 Anatomy-Consistency-Gated Fusion

The proposed fusion methods make image predictions the default while learning reliability-weighted metadata corrections, then structure prediction by clinical state and fracture location. Anatomy-consistency gating further attenuates corrections when reported bone type conflicts with image-side anatomy.

  • 2.7 Reliability-Gated Residual Fusion: Residual Fusion treats the image prediction as the default and learns a class-specific contextual correction rather than weighting image and metadata equally.A residual network predicts the correction, while a reliability network estimates the corresponding reliability signal.
  • 2.7 Reliability-Gated Residual Fusion: When metadata are unavailable, a = 0 makes the model fall back to the image branch; clinical modality dropout was applied during training with probability 0.25.The fallback applies when all metadata are unavailable.
  • 2.8 Hierarchical State–Location Prediction: The four target labels are separated into clinical state and acute-fracture location because distal and proximal denote location, whereas non-fracture and post-fracture denote state.This hierarchy reflects the semantic heterogeneity of the target labels.
  • 2.8 Hierarchical State–Location Prediction: Location probabilities are modeled only for acute fractures, while state and location probabilities reconstruct the original four-class distribution.The location loss is applied only to acute-fracture examples.
  • 2.9 Anatomy-Consistency-Gated Fusion: Residual reliability measures learned usefulness and input availability but does not explicitly detect a semantically wrong bone-type field.An image-side anatomy head was introduced to address this limitation.
  • 2.9 Anatomy-Consistency-Gated Fusion: Anatomy consistency compares the image-side predicted probability for the reported bone type, using floor α = 0.05 and exponent γ = 1.When bone type is unavailable, c = 1, so age, sex, and laterality are not suppressed.
  • 2.9 Anatomy-Consistency-Gated Fusion: The anatomy-consistency mechanism was evaluated in both flat and hierarchical variants, with the overall reliability- and anatomy-consistency-aware architecture summarized in Figure 1.The figure presents the proposed residual-fusion architecture.

2.10 Training Objectives · 2.11 Robustness Conditions · 2.12 Evaluation and Statistical Analysis

The study combined weighted classification, auxiliary, hierarchical, ranking, anatomy, and fallback objectives with controlled metadata corruption tests. Evaluation emphasized macro-F1, calibration, matched fold–seed comparisons, and cautious qualitative interpretation of Grad-CAM heatmaps.

  • 2.10 Training Objectives: Class-weighted cross-entropy with 0.05 label smoothing supervised four-class predictions, while auxiliary image, clinical, and anatomy heads received weights 0.25, 0.05, and 0.25.Hierarchical models also used state and acute-location losses.
  • 2.10 Training Objectives: Counterfactual ranking and safe-fallback objectives addressed shuffled-context failure modes, and the total loss combined classification, auxiliary, hierarchical, ranking, anatomy, and fallback terms.The ranking margin was m = 0.20.
  • 2.10 Training Objectives: AdamW training used backbone learning rate 10^-5, head learning rate 3 × 10^-4, weight decay 10^-4, batch size 16, cosine decay, mixed precision, and validation macro-F1 early stopping.The main experiment allowed up to 18 epochs with patience 4.
  • 2.11 Robustness Conditions: Robustness tests modified only test metadata while preserving images, using per-field missingness, complete removal, cross-patient shuffling, and bone-type replacement with valid alternatives.Missingness was tested at 25%, 50%, and 75%; bone type was replaced for 50% or 100% of examples.
  • 2.11 Robustness Conditions: Each stochastic robustness condition was repeated ten times in the five-fold targeted experiment, with validation-fitted temperature scaling applied to clean and corrupted predictions.Scaling enabled comparable Brier score and expected calibration error measurements.
  • 2.12 Evaluation and Statistical Analysis: Macro-F1 was primary for imbalanced, clinically distinct classes; accuracy, balanced accuracy, macro-AUROC, negative log-likelihood, Brier score, and ECE were also reported.Temperature scaling was fitted per validation fold, and the main five-fold, three-seed analysis summarized 15 runs with paired fold–seed differences.
  • 2.12 Evaluation and Statistical Analysis: Grad-CAM supported qualitative inspection, but heatmaps were not treated as quantitative localization evidence because OrthoFrac-XR lacks lesion bounding boxes.This limitation constrained interpretation of visual explanations.

3 Results

Permitted metadata contained substantial class information, and multimodal fusion improved fracture classification and calibration over image-only learning. Anatomy-consistency gating reduced performance losses under mismatched metadata but traded off clean-data performance, while auxiliary anatomy supervision helped when bone type was unavailable at inference.

  • Metadata signal: 0.4984 ± 0.0413 macro-F1 was achieved by metadata-only boosting, with bone type alone reaching 0.4401 ± 0.0388.Among permitted inputs, bone type had the largest association with the target at 0.113, followed by age at 0.062; laterality and sex were below 0.01.
  • Main multimodal comparison: 0.6046 ± 0.0279 macro-F1 was achieved by hierarchical residual fusion, versus 0.5727 ± 0.0270 for image-only ConvNeXt.Concatenation and residual fusion reached 0.6015 ± 0.0234 and 0.6027 ± 0.0214, respectively; hierarchical residual fusion improved macro-F1 by 0.0319 and Brier score by 0.0291 relative to image-only learning.
  • Bone type unavailable: 0.5899 ± 0.0289 macro-F1 with bone type excluded exceeded 0.5620 ± 0.0330 for ordinary residual fusion.The paired macro-F1 improvement was 0.0262, with a 95% bootstrap interval of approximately [0.0031, 0.0491].
  • Robustness to corrupted metadata: 0.0203 macro-F1 loss under shuffled structured fields was observed for anatomy-consistency fusion, versus 0.0567 for ordinary residual fusion.Under 100% wrong bone type, losses were 0.0183 and 0.0334 for anatomy-consistency and residual fusion, respectively, although absolute corrupted performance was similar.
  • Clean-performance trade-off: 0.41 to 0.53 was the mean anatomy-consistency probability for correctly matched all-metadata samples across folds.This moderate value contributed to the clean-data trade-off because the multiplicative gate attenuated metadata corrections even when context was correct.

4 Discussion

Leakage-safe structured context improved fracture classification, while anatomy-consistency gating reduced harm from mismatched metadata at a clean-performance cost. Interpretation is limited by the regional retrospective cohort, absent patient identifiers and external validation, metadata-only inputs, and one-seed consistency evaluation.

  • Main findings: Leakage-safe structured context improved fracture classification over image-only learning, with multimodal models gaining approximately three macro-F1 percentage points and reducing Brier score.Gains occurred in most matched runs across five folds and three seeds.
  • Main findings: Architectural complexity was not automatically beneficial: concatenation, residual fusion, and hierarchical residual fusion had nearly identical mean performance.Hierarchical image-only learning did not improve over the flat image baseline, so hierarchy was not independently responsible for the multimodal gain.
  • Robustness: 0.0203 macro-F1 loss occurred with anatomy-consistency fusion under shuffled metadata, versus 0.0567 for ordinary residual fusion.Under fully wrong bone type, the loss was approximately half that of residual fusion; the image branch remained primary while contradictory anatomy attenuated clinical corrections.
  • Auxiliary supervision: Auxiliary anatomical supervision improved the no-anatomy setting by encouraging the image encoder to organize features by skeletal structure without bone type at inference.The passage attributes this effect to multi-task representation learning and notes significant improvement of anatomy-consistency fusion over residual fusion in this setting.
  • Limitations: The regional benchmark is retrospective, heavily pediatric, and drawn from four institutions, while performance across hospitals, devices, adult populations, and countries remains unknown.The study also lacks patient-level identifiers, uses one seed for the anatomy-consistency extension, relies on structured metadata rather than symptoms, faces anatomical-shortcut concerns, uses retrospective labels without blinded re-reading, and provides qualitative rather than quantified localization.
  • Clinical trade-off: The strongest clean targeted model was concatenation, whereas consistency gating was safer under incorrect context, making metadata reliability a clinical model-selection consideration.Concatenation may be preferred when metadata are highly reliable, while consistency-aware fusion is more conservative when records may be mismatched or manually entered.

5 Conclusions

Leakage-safe structured metadata improved fracture classification over a strong image-only ConvNeXt baseline. Reliability-gated and hierarchical fusion delivered the strongest class-balanced and calibration results, while anatomy-consistency gating reduced degradation from mismatched metadata.

  • Leakage-safe structured metadata improved fracture classification over a strong image-only ConvNeXt baseline.
  • Reliability-gated residual and hierarchical fusion achieved the best mean class-balanced and calibration results in the full five-fold, three-seed experiment.Their clean discrimination was statistically close to simple concatenation.
  • An image-derived anatomy-consistency gate reduced degradation under shuffled and incorrect metadata.

Funding

The research received no external funding.

  • The research received no external funding.

Institutional Review Board Statement

This study was a secondary analysis of a publicly released, de-identified dataset, with no new patient recruitment or intervention; the original data collection had ethics approval under protocol REC-FSIT-2025/12639.

  • The analysis used a publicly released, de-identified dataset and involved no new patient recruitment or intervention.

Informed Consent Statement

The secondary analysis relied on consent procedures and de-identification managed by the original dataset creators, with no identifiable information accessed.

  • Informed Consent Statement: The original dataset creators managed patient consent procedures and de-identification for the study data.No identifiable information was accessed during the present secondary analysis.

Abbreviations

This section defines the manuscript’s abbreviations for artificial intelligence, performance evaluation, calibration, divergence, and neural-network architecture.

  • AI denotes artificial intelligence, and MLP denotes multilayer perceptron.
  • AUROC denotes area under the receiver operating characteristic curve, while ECE denotes expected calibration error.
  • KL denotes Kullback–Leibler.
Loading 2608.21482v1…