PaperlayerView exact revision on arXiv

Source-linked AI summary

FleXray: Universal Clinical X-ray Segmentation

Victor Ion Butoi, Vivek Gopalakrishnan, John V. Guttag, Adrian V. Dalca, Neel Dey

arXiv:2609.26756v2cs.CVcs.AI

TL;DR

Analytical DRRs do not capture the full clinical appearance of X-rays, while broad segmentation must cover anatomical, geometric, and appearance variation. FleXray combines physics-based projections, generative editing, and targeted real X-rays; it generalizes across held-out datasets and supports quantitative workflows, though its evaluation and training sources leave important coverage gaps.

  • Problem

    Analytical DRRs do not reproduce the full appearance distribution of clinical X-rays, including text overlays and higher-order effects such as scatter.

    Source: Results · p. 3

  • Method

    FleXray combines physics-based analytic projections, quality-controlled generative editing, and targeted real X-rays to train full-body segmentation networks.

    Source: Results · pp. 3, 5

  • Results

    FleXray matched or exceeded prior methods across held-out segmentation datasets and approached dataset-specific custom networks, with performance depending on broad training-distribution coverage.

    Source: Discussion · p. 10

  • Takeaways & Limitations

    Its anatomical predictions support disease-grading measurements, semantic guidance for 2D/3D registration, and data-efficient learning for unseen targets.

    Source: Discussion · p. 10

    Evaluation ground truth is available for few datasets, and CT-source biases plus unsynthesized pathologies and hardware leave possible failures on unseen acquisition geometries.

    Source: Discussion · p. 12

Abstract

from arXiv · show

X-ray is medicine's most widely used imaging modality, yet remains among its least quantitative. Unlike volumetric modalities like CT or MRI, X-ray collapses 3D anatomy into a 2D projection, causing structures to overlap and anatomical boundaries to be ambiguous, even to experts. As a result, labeling X-ray databases for training general-purpose segmentation systems is impractical, leaving morphometric and functional X-ray analysis confined to narrow anatomical regions and applications. To this end, we present FleXray, a generalist model for anatomical segmentation across the entire body in clinical X-rays. Instead of curating large, manually annotated X-ray datasets, we build a scalable, physics-based generative X-ray data engine. Using existing 3D whole-body CT segmentation datasets and generative image-editing models, we simulate fully-annotated 2D X-rays with diverse appearances, physiological properties, and imaging geometries. Trained on these simulations, FleXray accurately segments 60 anatomical structures across unseen research datasets and in-the-wild X-rays. We further show that FleXray makes X-rays directly amenable to quantitative analysis, enabling automated measurements for disease grading, robust navigation during X-ray-guided interventions, and data-efficient learning of pathological targets. We release the model, code, a full-body X-ray segmentation dataset, and a local, easy-to-use browser-based tool at https://flexray.csail.mit.edu .

Results

FleXray combines simulated and targeted real X-rays to learn full-body segmentation, then generalizes across held-out datasets and supports downstream measurement, registration, and low-label adaptation.

Data engine and training

  • The data engine combines CT-derived analytic projections, quality-controlled generative editing, and targeted real X-rays, mapping heterogeneous annotations to 60 structures.Training preserves overlapping labels, routes losses according to source annotation coverage, and uses an ensemble with test-time augmentation; about 4% of enhanced DRRs are rejected by quality control.Source: Results · pp. 3, 5

Quantitative segmentation evaluation

  • Across eight held-out X-ray datasets, FleXray outperformed every supported baseline on seven; significance was p<0.001 on six, while VinDr-Rib was not significantly different from PAXray (p=0.146).FleXray averaged 0.875 Dice versus 0.914 for in-domain nnU-Nets, trailing by 0.039 on average. On DeepFluoro, it outperformed the dataset-trained nnU-Net, 0.915 versus 0.866.Source: Results · p. 5, Figure 3 · p. 6

Robustness to acquisition conditions

  • On DeepFluoro, mean Dice was 0.915 across oblique views; FleXray also produced coherent segmentations on varied in-the-wild Radiopaedia images.Lumbar-spine Dice declined at larger angles, while femurs, hips, and sacrum remained within 0.05 Dice of frontal-view performance.Source: Results · p. 5, Figure 4 · p. 7

Ablating generative enhancement

  • Generative enhancement left average Dice unchanged but improved VinDr-Rib Dice by 3.9 points pooled across ribs 5–10, with larger gains on lower ribs.The reported gains increased from 2.5 points at rib 5 to 7.4 at rib 10.Source: Figure 4 · p. 7, Results · p. 8

Ablating attenuation randomization

  • Attenuation randomization raised pneumonia lung Dice from 0.894 to 0.914 and reduced pneumonia images below 0.85 Dice from 122 to 50.The gain was negligible in healthy lungs, where Dice changed from 0.945 to 0.946.Source: Figure 4 · p. 7, Results · p. 8

Scoliosis measurement

  • FleXray measured Cobb angles on all 218 AASCE images, with 5.13° mean absolute error, below expert inter-observer variability.It achieved the lowest error in each severity stratum; 61% of estimates fell within 5° and 88% within 10° of reference.Source: Results · p. 8, Figure 5 · p. 9

2D/3D registration

  • Adding FleXray-derived Chamfer loss reduced median registration error from 324.5 mm to 1.2 mm from a manual frontal start and reduced gross failures to 12.7%.With foundation-model initialization, it recovered all eight cases above 50 mm and reduced gross failure rate to 3.3%.Source: Results · pp. 9–10, Figure 5 · p. 9

Adaptation to unseen segmentation tasks

  • With 10% of BTXRD training data, finetuned FleXray reached 23% AP50 versus 7% for nnU-Net; its initialization benefit was largest with scarce labels.On MTDDH with 1% of training data, FleXray reached 0.896 mean Dice versus 0.867 for nnU-Net.Source: Results · p. 10

Discussion

FleXray generalizes across held-out X-ray datasets, and its predictions support quantitative workflows beyond segmentation. The discussion highlights training-distribution coverage, reusable resources, and remaining scope limitations.

Scaling generalist X-ray segmentation beyond manual annotations

  • Across held-out segmentation datasets, FleXray matched or exceeded prior methods without label-set-specific models or per-structure prompts, and approached dataset-specific custom networks.The paper reports evaluation across eight datasets spanning the limbs, pelvis, spine, and chest.Source: Discussion · pp. 10, 12
  • Appearance randomization accounts for most of the gain; generative enhancement helps where analytic rendering is least faithful, while limb X-rays close CT coverage gaps despite only 4% of training iterations.These findings support the discussion’s emphasis on broad coverage of anatomical, geometric, and appearance variation at deployment.Source: Discussion · p. 10

Making X-rays a quantitative modality

  • FleXray supports disease-grading measurements, robust 2D/3D registration, and data-efficient learning of unseen targets, with initialization most useful when labels are scarce.In registration, semantic predictions remove catastrophic optimization failures (mTRE >50 mm) without degrading successful registrations.Source: Discussion · p. 10

A reusable resource for full-body X-ray analysis

  • The released resource includes enhanced DRRs, manual forearm and humerus annotations, seven redistributable real-X-ray sources, and fixed evaluation splits for comparable testing.The authors expect the resource to lower the barrier to developing pan-anatomy X-ray models beyond the architecture studied here.Source: Discussion · p. 10

Making X-rays a quantitative modality

  • Figure 6 evaluates transfer across annotation budgets for MTDDH segmentation and BTXRD tumor detection, including Dice, AP50, and tumor-size strata.Its caption describes budget fractions from 1% to 100%; it supplies the evaluation setup, not numerical outcome values.Source: Figure 6 · p. 11

Limitations and future work

  • Evaluation ground truth remains limited, and CT-source biases plus unsynthesized pathologies, implants, and surgical hardware leave potential failures on unseen geometries and severe out-of-distribution cases.The authors note that none of their quantitative experiments include datasets with substantial clinical text overlays.Source: Discussion · p. 12

Methods

The training data engine combines online CT projections, diffusion-enhanced synthetic X-rays, and annotated real X-rays, then harmonizes their labels into a common 60-structure space.

A Training data engine

  • Training draws from online CT-rendered DRRs, diffusion-enhanced synthetic X-rays, and annotated real X-rays, with sources sampled using mixture weights.Online MOOSE and enhanced-DRR sources together account for 75% of the sampling budget; other sources provide the remaining 25%.Source: A Training data engine · p. 18, Table 1 · p. 19

A.1 DRRs generated online from CT scans

  • Online DRRs integrate CT attenuation along randomized views, while projected labels use ray attenuation and a 0.1 lung threshold to focus masks on visible lung fields.Per-label attenuation is also randomized independently to simulate organ-density variation.Source: §A.1 · p. 18, Figure 7 · p. 18

A.2 Diffusion-enhanced synthetic X-rays

  • Diffusion editing adds clinical appearance to MOOSE DRRs while retaining projected labels; pose-specific paired SoftDice filtering removes samples with excessive segmentation degradation.The upper-tail cutoff mirrors the empirical second percentile around the pose-group median.Source: §A.2 · pp. 18–20

A.3 Annotated real X-rays

  • Annotated hand, foot, forearm, and humerus X-rays supplement CT supervision for peripheral bones and significantly improve held-out wrist and elbow performance.Source: §A.3 · p. 20

A.4 Label harmonization

  • An automatic harmonization pipeline maps source-specific labels into 60 structures represented in radiographs, merging finer distinctions and dropping excluded structures from supervision.The vocabulary omits anatomy generally invisible on X-ray.Source: §A.4 · p. 20

A.5 Data augmentation

The pipeline augments images with X-ray-specific transformations and standardizes data and splits for training and evaluation.

A.5 Data augmentation

  • Training augmentations include structure-centered Label Zoom, randomly placed laterality markers, and aspect-ratio crops, alongside other image and mask transforms.The augmentation presets specify which operations affect images, masks, or particular data streams.Source: §A.5 · pp. 20–21, Table 3 · p. 22
  • Pose-group filtering removes high-∆ Flux-enhanced samples, and the surviving examples show substantially reduced structural hallucination.The figure contrasts raw-minus-enhanced SoftDice differences and worst-case samples before and after filtering.Source: Figure 8 · p. 21, §A.5 · p. 21

A.6 Preprocessing

  • CT volumes are stored as overlapping crops for online rendering, while X-rays are padded and resized to 256×256 and subject-disjoint splits are fixed across experiments.Images are intensity-clipped to per-image percentiles and rescaled to [0, 1].Source: §A.6 · p. 21

B.1 Network architecture

FleXray uses an ensemble trained with mixed data and coverage-aware losses, then evaluates on held-out radiographs using dataset-specific scoring adjustments.

B.1 Network architecture

  • The final architecture ensembles five seven-level 2D U-Nets whose members differ only in their training-data mixtures.Each network predicts independent, potentially overlapping structure masks.Source: §B.1 · p. 22

B.2 Optimization

  • Crop sampling weights favor underrepresented labels, and training equally combines Soft-Dice and binary crossentropy losses.Soft-Dice applies to present foreground channels, while BCE supervision is adjusted for partially labeled sources.Source: §B.2 · p. 22
  • For partially labeled sources, unannotated channels are excluded from BCE; anatomically impossible structures receive targeted negative supervision to limit extra-class hallucinations.Source: §B.2 · pp. 22–23

C Evaluation

  • Evaluation uses eight held-out X-ray datasets spanning limbs, pelvis, spine, and chest, with test partitions reserved from model development.The datasets cover varied acquisition settings and adult and pediatric anatomy; models are evaluated at their final training epoch without checkpoint selection.Source: C Evaluation · pp. 23–24

C.1 Protocol post-processing

  • Dataset-specific post-processing aligns predictions with annotation protocols, including lung masking, vertebral-channel aggregation, and exclusive overlays where required.Rules were fixed before scoring and applied identically to every method.Source: §C.1 · p. 24

C.2 Metrics

The paper defines segmentation, hallucination, registration, and detection metrics, then describes test-time augmentation and evaluation protocols for baselines, Cobb-angle estimation, and 2D/3D registration.

C.2 Metrics

  • Dice measures mask overlap, HD95 measures boundary error in pixels, and aFPR counts predicted classes incompatible with the dataset anatomy.HD95 excludes structures absent from thresholded predictions; structures occupying less than 0.1% of an image are omitted from its score.Source: §C.2 · p. 24
  • mTRE reports mean landmark registration error in millimeters, while AP50 scores lesion localization using an intersection-over-union threshold of 0.5.AP50 is the area under the 101-point interpolated precision–recall curve.Source: §C.2 · pp. 24–25

C.3 Test-time augmentation

  • TTA averages predictions across stochastic views, beginning with the unaugmented image; the preset applies independent augmentations and inverse alignment handles left/right flips.K = 16 samples was fixed using validation partitions before final training, then swept on test partitions to show metric variation.Source: §C.3 · p. 25, Table 5 · p. 25
  • 16-sample TTA lowers aFPR from 3.64 to 2.35 classes per image for FleXray (single); ensembling plus TTA yields 1.18.Ensembling alone lowers aFPR to 1.43; equal-dataset mean Dice rises from 0.866 to 0.875 and HD95 falls from 11.8 to 10.8 pixels between FleXray (single) and FleXray.Source: §C.3 · p. 25

C.4 Baselines

  • Baselines are evaluated on released output labels, with method-specific TTA; the efficiency protocol times native-resolution inference and includes augmentation and averaging where applicable.Dataset-specific nnU-Net comparators use default settings without dataset-specific hyperparameter tuning or fold ensembling.Source: §C.4 · pp. 25–26

D.1 Cobb-angle estimation

  • Cobb angles are estimated from vertebral masks by ordering vertebrae and deriving end-plate directions from adjacent spinal centerline segments.The largest pairwise angular separation is the predicted major Cobb angle; images with fewer than three surviving vertebrae count toward failure rate.Source: §D.1 · p. 26

D.2 2D/3D registration

  • Registration adds bidirectional Chamfer terms between FleXray pelvic masks and structure-specific CT projections to xvr’s image-similarity objective.Chamfer weights are scheduled by optimization stage, with stronger coarse-stage weights for coarse initialization and weaker weights for foundation-model initialization.Source: §D.2 · pp. 26–27
  • Distance-transform Chamfer terms provide informative gradients even when DRR and X-ray images no longer overlap, and evaluation compares registrations with and without those terms.The evaluation uses two held-out DeepFluoro specimens and two initializations; registration is successful below 10 mm mTRE.Source: §D.2 · p. 27

D.3 Data-efficient finetuning

The section describes fine-tuning FleXray for target protocols and comparing it with nnU-Net across training-data budgets, with dataset-specific label and evaluation choices.

D.3 Data-efficient finetuning

  • Fine-tuning initializes from FleXray (single), replaces its output convolution with a randomly initialized target-sized head, and trains end-to-end.Runs use a fixed 250-epoch schedule and select checkpoints by validation foreground Dice.Source: §D.3 · p. 27
  • Few-shot subsets use three random-seed repeats below the full-data budget, and FleXray and from-scratch nnU-Net use identical subset manifests and splits.Within each repeat, smaller subsets are prefixes of larger ones.Source: §D.3 · p. 27
  • MTDDH fine-tuning uses four foreground labels with patient-level budgets, while BTXRD uses stratified image subsets and tumor AP50 evaluation.BTXRD contains tumors in every image, so precision does not estimate specificity on normal radiographs.Source: §D.3 · p. 28

E.1 Full quantitative quality comparison against baselines

Across held-out datasets, FleXray often outperforms generalist baselines, though in-domain nnU-Nets lead on most datasets; the single FleXray model retains nearly the released configuration’s accuracy at much lower latency.

E.1 Full quantitative quality comparison against baselines

  • FleXray exceeds FluoroSAM by 0.42–0.71 Dice on wrist, elbow, and lower-limb datasets, despite FluoroSAM receiving ground-truth-derived prompts and clicks.On VinDr-Rib, PAXray’s 0.751 Dice is not significantly different from FleXray’s 0.737 (p=0.146); FleXray leads TotalSegmentator2D on PedsTorso and DarwinCVD19.Source: §E.1 · p. 28
  • In-domain nnU-Nets exceed FleXray on seven datasets by 0.03–0.09 Dice, while FleXray scores 0.915 versus 0.866 on held-out DeepFluoro subjects.The DeepFluoro comparison uses 213 frames and reports p<0.001.Source: §E.1 · p. 28
  • FleXray (single) without TTA retains 99% of the released configuration’s accuracy at approximately 1/84 of its latency.The released five-member ensemble takes 74.2 ms per image, while its TTA configuration takes 1.24 s.Source: §E.1 · p. 28

E.2 Ablations

The ablation study evaluates versions of FleXray (single) to isolate components of its training pipeline on held-out real-X-ray data.

E.2 Ablations

  • The ablation variants are designed to evaluate different parts of the FleXray training pipeline.They use the FleXray training recipe except for the components under study.Source: §E.2 · p. 28
  • Ablations use SAM2-specific input resolutions and optimization settings where applicable.Source: §E.2 · p. 28
  • All ablation models are tested on the held-out real-X-ray suite using the same 16-sample TTA.Source: §E.2 · p. 28

E.2.1 Mixing online and offline rendering performs best

Balanced online and offline rendering performed best among the tested mixtures. In the same evaluation, finetuned SAM2 encoders showed no statistically detectable accuracy advantage over FleXray’s from-scratch U-Net.

E.2.1 Mixing online and offline rendering performs best

  • 0.870–0.874 equal-dataset mean Dice was reached by the three mixed rendering splits; the selected 37.5/37.5 split represents FleXray (single).Both rendering extremes performed worse: online-only scored 0.858 and offline-only 0.852.Source: §E.2.1 · pp. 28–29

E.2.2 Finetuned SAM2 encoders show no significant advantage over a U-Net

  • SAM2 Hiera encoders were fully finetuned with a linear segmentation head, while the comparison shared FleXray’s data engine, label harmonization, augmentations, objective, and evaluation pipeline.The four tested encoder sizes were Tiny, Small, Base+, and Large.Source: §E.2.2 · p. 29
  • 0.876 [0.829, 0.916] mean Dice for Hiera Large was not significantly different from FleXray (single) at 0.870 [0.817, 0.915].The paired difference was 0.006 [−0.009, 0.023], an interval that includes zero.Source: §E.2.2 · p. 29

E.2.3 Image-space augmentation drives transfer to real X-rays

The augmentation ablation shows that transfer to real X-rays improved as appearance variation was added, with the complete preset yielding the highest reported mean Dice. Inversion recovered much, but not all, of that gain.

E.2.3 Image-space augmentation drives transfer to real X-rays

  • Adding intensity inversion improved seven of eight datasets, with the largest gains on HipRay (+0.46) and DeepFluoro (+0.39); the eight-image LowerLimbs set was the exception.On rib and pediatric datasets, the inversion regime already matched the complete preset.Source: §E.2.3 · p. 30
  • Attenuation randomization alone improved only DarwinCVD19 (+0.15 Dice) and DeepFluoro (+0.07), while the other six datasets were indistinguishable from the base regime.Only the complete preset recovered the wrist and elbow datasets, which remained below 0.3 Dice under every partial regime.Source: §E.2.3 · p. 30
  • 0.353, 0.438, 0.650, and 0.870 equal-dataset mean Dice were reported for the base, attenuation-only, inversion-added, and complete-preset regimes, respectively.The complete preset is FleXray (single) and was retained in the released model.Source: Figure 11 · p. 30, §E.2.3 · p. 30

E.2.4 Real X-rays provide training coverage for peripheral radiographs

A training-source ablation found that real X-ray annotations improved wrist and elbow segmentation, while supplemental CT added a smaller elbow gain. Out-of-suite examples also expose gaps in represented acquisition conditions and anatomical variation.

E.2.4 Real X-rays provide training coverage for peripheral radiographs

  • 0.712 to 0.870 equal-dataset mean Dice accompanied inclusion of 535 real X-ray training radiographs; RAM-W600 rose from 0.004 to 0.946 and ElbowLat gained +0.325 Dice.These source effects were significant; Table 9 reports the ablation on RAM-W600 and ElbowLat.Source: §E.2.4 · p. 31
  • The ablation compared mixtures omitting source families; supplemental CT added +0.024 Dice on ElbowLat, while skull and shoulder contributions were not evaluated by the held-out suite.No other dataset met both the significance and 0.01-Dice criteria in either source comparison.Source: §E.2.4 · pp. 30–31

F Failure cases

  • Six radiographs outside the evaluation suite illustrate limitations spanning oblique or lateral views, collimation and exposure, pediatric anatomy, and surgical hardware.The authors connect these examples to gaps in training-source coverage and motivate additional targeted data.Source: F Failure cases · p. 31

H Data availability

The data-availability materials list public CT and X-ray sources and describe which annotations and synthetic radiographs are released. They also include out-of-suite failure examples and report the authors’ contributions.

H Data availability

  • Listed public CT sources include ElbowCT, HaN-Seg, MOOSE, PedsCT, RSNAFrac, and Shoulder-CT.The materials also list additional public CT datasets.Source: H Data availability · p. 31
  • Listed public X-ray sources include DarwinCVD19, DeepFluoro, ElbowLat, FootBones, HandBones, and HipRay.The materials also list LowerLimbs, MTDDH, RAM-W600, PedsTorso, VinDr-Rib, and BTXRD.Source: H Data availability · pp. 31–32
  • Figure 12 presents six radiographs outside the evaluation suite, including peripheral views, pediatric anatomy, hardware, and a heavily collimated, overexposed shoulder.Panels include MURA validation images not annotated for training and Radiopaedia images.Source: Figure 12 · p. 32
  • Redistributable X-ray datasets, split assignments, quality-control exclusions, and specified MURA annotations are shared alongside 138,063 enhanced DRRs under CC BY-NC 4.0.The DRRs’ license is inherited from the AutoPET-derived subset of the MOOSE source CTs.Source: H Data availability · pp. 32–33

J Author Contributions

  • V.I.B. led code, data preparation, training, experiments, figures, and manuscript development; V.G. developed the rendering toolkit and contributed datasets and a registration experiment.N.D. conceived the initial idea, identified training and evaluation datasets, and provided technical direction.Source: J Author Contributions · p. 33

K.1 Statistics and reproducibility

The appendix specifies statistical uncertainty and testing conventions, implementation setup, public-dataset anatomical-coverage accounting, and attribution handling for Radiopaedia examples.

K.1 Statistics and reproducibility

  • Results are means with 95% percentile-bootstrap confidence intervals from 10,000 resamples, with resampling units tailored to each analysis.Source: §K.1 · p. 34

K.2 Implementation details

  • Each ensemble member trained on one NVIDIA H200 for approximately 60–70 hours, with roughly 320 GPU-hours for the ensemble; projections used nanoDRR.Source: §K.2 · p. 34

K.3 Anatomical coverage of public X-ray segmentation data

  • Public-dataset coverage counts images carrying masks per structure, collapses the 60-structure protocol to 25 display regions, and excludes distal radius and ulna coverage.Source: §K.3 · p. 34, Table 10 · p. 34

K.4 Radiopaedia image attributions

  • Radiopaedia examples in Figures 4 and 12 are used under CC BY-NC-SA 3.0, with each source X-ray and its segmentation overlay sharing an attribution.Source: §K.4 · p. 34

Related papers

Loading 2609.26756v2…