PaperlayerView exact revision on arXiv

Source-linked AI summary

Abdominal Ultrasound Simulation from Semantic Labels using Paired Label-to-Physics-Based Image Translation

Santiago Vitale, Duilio Deangeli, Ignacio Larrabide, José Ignacio Orlando

arXiv:2610.08849v1eess.IVcs.CV

TL;DR

Abdominal US simulation remains dependent on CT data or precomputed simulations, limiting direct synthesis from semantic anatomical masks. This paper trains paired label-to-physics synthesis from CT-ray-casting supervision and refines its output with SG-CycleGAN; SDM favors morphological consistency, while Pix2Pix favors perceptual similarity and lower inference latency.

  • Problem

    Existing abdominal US synthesis relies on CT images, CT-derived simulations, or precomputed images, while direct synthesis from dense anatomical masks remains underexplored.

    Source: §1 · pp. 2–3

  • Method

    The framework trains Stage I to map semantic masks to CT-derived ray-casting images, then uses SG-CycleGAN as a downstream unpaired refinement stage.

    Source: §1 · p. 3, §4 · pp. 19, 21

  • Results

    SDM better preserved spatial and anatomical structure, whereas Pix2Pix better matched perceptual and distributional characteristics and had substantially lower inference latency.

    Source: §4 · p. 21

  • Takeaways & Limitations

    Modified semantic labels produced qualitatively plausible proof-of-concept US appearances for liver tumors, abdominal aortic aneurysms, kidney stones, and gallbladder polyps.

    Source: §4 · p. 21

    Training still depends on plausible segmentations and ray-casting simulations, and the relatively small number of CT volumes limits represented variability and assessment of generalizability.

    Source: §4 · p. 22

Abstract

from arXiv · show

Purpose: Current abdominal ultrasound (US) simulation methods often require CT-based anatomical references for ray-casting, limiting deformation and pathology variability. We propose a learning-based pipeline trained to predict physics-based images derived from CT scans from semantic labels, enabling controlled simulations without patient-specific CT volumes at inference time. Methods: We introduce a two-stage pipeline that maps anatomical segmentations to realistic US images through a simplified US image. Stage~I synthesizes this image from semantic labels using models trained on CT-based ray-casting outputs. Stage~II refines it into a realistic US scan using anatomically guided unpaired translation. Deformations and pathologies are generated by editing anatomical maps. Results: We evaluated Pix2Pix and the Semantic Diffusion Model (SDM) in Stage~I, followed by segmentation-guided CycleGAN (SG-CycleGAN) refinement in Stage~II. SDM significantly outperformed Pix2Pix in morphological metrics, including MAE (19.85 vs. 21.65), SSIM (0.28 vs. 0.24), and mIoU (0.43 vs. 0.29), whereas Pix2Pix yielded better perceptual point estimates (LPIPS: 0.17 vs. 0.19; FID: 0.32 vs. 0.37; KID: 0.25 vs. 0.48). Conclusion: Training paired generative models with physics-based supervision enables approximation of CT-derived ray-casting outputs at inference time directly from semantic labels. Although the pipeline does not require patient-specific CT volumes at inference time, CT-derived segmentations and ray-casting simulations remain necessary to train Stage~I. Once trained, the framework enables controllable healthy and pathological simulations through semantic-map modification.

1 Introduction

Multi-organ abdominal US synthesis from semantic masks remains underexplored, while existing approaches depend on CT data, simulated images, or restricted anatomical settings. This work trains a label-driven pipeline with CT-derived ray-casting supervision, then uses label edits to support controlled scenario generation without a new patient-specific scan at inference.

1 Introduction

  • The framework generates abdominal US from semantic masks, using CT-derived ray-casting simulations for paired training and SG-CycleGAN as a downstream refinement module.The work evaluates whether new Stage I outputs remain compatible with the previously established refinement strategy.Source: §1 · p. 3
  • Existing abdominal US synthesis relies on CT images, CT-derived simulations, or precomputed simulated images, and dense-label paired datasets are scarce.Multi-organ synthesis also faces large fields of view, interacting organs, acoustic shadowing, occlusions, view-dependent artifacts, and high anatomical variability.Source: §1 · p. 3
  • The paired Stage I models approximate CT-derived ray-casting outputs during inference without executing ray-casting or requiring a new patient-specific CT or MRI volume.The proposed method still requires CT-derived ray-casting simulations as paired supervision during training.Source: §1 · p. 3
  • Semantic-map edits support proof-of-concept scenarios involving anatomical deformations and synthetic liver tumors, kidney stones, and gallbladder polyps.Optional label transformations also include motion and lesions.Source: §1 · p. 4, Figure 1 · p. 4

2 Materials and Methods

The pipeline first pairs anatomical masks with simplified physics-based US images generated from CT scans, then maps masks to those images and refines them toward realistic US. At inference, label-map transformations can introduce motion, lesions, or deformations.

2 Materials and Methods

  • A ray-casting simulator generates simplified 2-D US images from CT scans and their associated segmentations, forming paired mask–image training data.The resulting paired dataset is used to train a generator that maps segmentation masks to synthetic US slices.Source: §2 · p. 4
  • Stage I maps semantic masks to simplified US images, and the unpaired SG-CycleGAN Stage II refines realism while preserving anatomy.The two generators are trained independently; the refiner adds speckle texture, scattering effects, and artifacts absent from the intermediate images.Source: §2 · p. 4, Figure 2 · p. 5
  • At inference, transformed input labels enable simulation of US sequences with breathing motion, synthetic lesions, or operator-induced abdominal-pressure deformations.The pipeline overview also describes label transformations such as motion, lesions, and deformations without CT input.Source: §2 · p. 5

2.1 Data preparation

The dataset pairs abdominal anatomical masks with simplified physics-based ultrasound images, using public CT datasets, manual tumor annotations, and volunteer scans for the synthetic and real-image domains.

2.1.1 Materials

  • 39 CT scans came from VISCERAL Anatomy3 and IRCAD, while the real-US domain comprised 617 B-mode images from 11 healthy volunteers.The IRCAD cohort included healthy and diseased subjects, with liver tumors in 41% of cases.Source: §2.1.1 · p. 5

2.1.2 CT segmentation

  • CT volumes were segmented into 16 anatomical structures with TotalSegmentator, supplemented by released manual liver-tumor annotations for IRCAD cases.The same discrete label convention and color mapping were used across model inputs and figures.Source: §2.1.2 · pp. 5–6

2.1.3 Generation of simplified US images with a physics-based simulator

  • A ray-casting simulator extracts a probe-defined 2D slice and simulates ultrasound-wave interactions at tissue interfaces to generate simplified US images.The simulation uses CT-derived Hounsfield Units and corrects tissue transmissions using liver values as a reference.Source: §2.1.3 · p. 6

2.1.4 Final dataset assembly

  • The dataset contains 1,036 paired mask–synthetic-image samples, split by CT volume into 711 training, 80 validation, and 245 test samples.Samples used polar coordinates; Pix2Pix generated 256 × 256 images, while SDM generated 128 × 128 images that were resized for comparison.Source: §2.1.4 · p. 6

2.2 Paired segmentation to physics-based image translation

Stage I learns paired translation from anatomical masks to simplified physics-based US images. Pix2Pix and SDM provide complementary conditional-generation approaches for this mapping.

2.2 Paired segmentation to physics-based image translation

  • Stage I maps segmentation masks to paired synthetic US images, comparing Pix2Pix with the diffusion-based SDM as alternative generative paradigms.The comparison targets perceptual realism, anatomical fidelity, and computational cost.Source: §2.2 · pp. 6–7

2.2.1 Pix2Pix

  • Pix2Pix was trained to condition physics-based ultrasound image generation on anatomical label maps using adversarial and reconstruction losses; at inference, G1 generates images from unseen maps without CT volumes or ray-casting.The discriminator follows a 70×70 PatchGAN design.Source: §2.2.1 · pp. 7–8

2.2.2 Semantic Diffusion Model (SDM)

  • SDM corrupts physics-based images with Gaussian noise and learns stepwise noise prediction, injecting anatomical labels into decoder features through SPADE blocks.At inference it starts from pure noise and iteratively denoises under label guidance; its ADM-style U-Net predicts the added noise.Source: §2.2.2 · p. 8

2.3 Unpaired physics-based to realistic image translation

Stage II refines Stage I synthetic images toward real ultrasound with an unpaired, segmentation-guided CycleGAN. Its segmentation-consistency term encourages preservation of organ geometry during translation.

2.3 Unpaired physics-based to realistic image translation

  • SG-CycleGAN extends CycleGAN with a segmentation-consistency term to encourage preservation of organ geometry during unpaired translation.It maps between the Stage I synthetic-image domain and real-US domain using two generators and discriminators.Source: §2.3 · p. 9
  • Unpaired synthetic and real scans feed the generators, while a pretrained segmenter supplies masks for the structural-alignment loss; inference uses only the synthetic-to-real generator.The refinement does not require paired real-US data.Source: §2.3 · p. 9
  • The segmenter was trained independently of Pix2Pix and SDM using simplified physics-based images and CT-derived masks, not either Stage I model’s generated images.Source: §2.3 · p. 10

2.4 Inference

At inference, the pipeline transforms an anatomically plausible semantic map into a realistic US image through two generators. Map edits provide the inputs for controlled deformations and pathological examples.

2.4 Inference

  • Inference applies G1 to the label map, then G2 to the simplified image, and resamples the polar-coordinate output to Cartesian space.Inputs use the training label convention and 256 × 256 polar representation.Source: §2.4 · p. 10
  • Deformations and pathologies are introduced by editing semantic maps; the breathing-like motion is a simplified model, and pathological examples are manually designed proof-of-concept cases.The deformation warps labels over eight temporal states while keeping bone fixed; pathology examples include liver tumors, aortic aneurysms, kidney stones, and gallbladder polyps.Source: §2.4 · p. 10

2.5 Evaluation metrics and statistical analysis

The evaluation combines complementary measures of reconstruction, anatomy, perception, and distribution similarity, with distinct protocols for the two pipeline stages and for statistical testing.

2.5 Evaluation metrics and statistical analysis

  • Stage I uses paired-reference morphological metrics for reconstruction and anatomical structure, alongside perceptual and distributional metrics for feature and set-level similarity.mIoU compares segmentations predicted for reference and generated images and is interpreted as relative segmentation consistency, not an independent estimate of anatomical accuracy.Source: §2.5 · p. 11
  • FID and KID compare feature distributions using an ImageNet-pretrained Inception v3 representation, with intermediate 768-dimensional features used for evaluation.KID is described as an unbiased estimator that can be more appropriate for relatively small datasets.Source: §2.5 · pp. 11–12
  • Stage II is evaluated for similarity to real US rather than paired reconstruction, using FID/KID, organ-region χ2 histogram distances, anatomy-aware measures, and qualitative inspection.No corresponding real US image was available for each refined image; the numeric metrics do not directly measure artifact correctness or clinical plausibility.Source: §2.5 · p. 12
  • Stage I image-level metrics use paired one-tailed Wilcoxon tests, while FID and KID use 95% bootstrap confidence intervals from 1,000 resamples.Dataset-level FID and KID are not subjected to image-level hypothesis testing.Source: §2.5 · p. 12

3 Results

The results section evaluates Pix2Pix and SDM in Stage I and tests how their outputs integrate with the Stage II refiner, using quantitative and qualitative assessment.

3 Results

  • Stage I compares Pix2Pix with SDM, which was run with 750 inference steps selected as a cost-effective trade-off between generation time and image quality.The evaluation also assesses refinement of outputs from each Stage I model.Source: §3 · p. 12
  • The Stage II integration assessment applies the refinement model to outputs from both Stage I methods.Models were implemented in PyTorch 2.0.1 and evaluated on a workstation with one NVIDIA Quadro P6000 GPU.Source: §3 · p. 12
  • Quantitative and qualitative evaluations were conducted for both stages on a desktop workstation running Ubuntu 18.04.6.The system had an Intel Core i7-10700K CPU and 16 GB of RAM.Source: §3 · p. 12

3.1 Quantitative Evaluation

The quantitative evaluation finds a Stage I trade-off between SDM’s morphological and anatomical scores and Pix2Pix’s perceptual and distributional scores; refinement improves distributional alignment, while speed favors Pix2Pix.

3.1.1 Stage I: Simplified US Simulation

  • MAE was 19.85 vs. 21.65, SSIM was 0.28 vs. 0.24, and mIoU was 0.43 vs. 0.29 for SDM vs. Pix2Pix, while Pix2Pix had lower LPIPS (0.17 vs. 0.19), FID (0.32 vs. 0.37), and KID (0.25 vs. 0.48).SDM’s morphological advantages included lower MSE (769.18 vs. 884.72) and higher PSNR (19.43 vs. 18.81); the FID confidence intervals overlapped, while KID intervals did not.Source: Table 1 · p. 13, §3.1.1 · p. 13

3.1.2 Stage II: Image Refinement

  • Stage II lowered FID from 1.66 to 0.39 and KID from 4.92 to 0.30 for Pix2Pix inputs, and from 2.09 to 0.53 and 6.28 to 0.84 for SDM inputs.Pix2Pix-refined outputs had lower FID (0.39 vs. 0.53) and KID (0.30 vs. 0.84) than SDM-refined outputs; organ-level echogenicity changes were smaller and region-dependent, and refined SDM liver outputs differed significantly from real data.Source: §3.1.2 · pp. 14–15, Table 2 · p. 15

3.1.3 Computational Cost and Inference Speed

  • Pix2Pix reached 155.25 FPS, and the complete Pix2Pix+SG-CycleGAN pipeline reached 63.33 FPS, whereas SDM took 27.87 s and 37.13 s per image at 750 and 1000 steps.SG-CycleGAN alone reached 130.54 FPS; 750 SDM steps were selected as a practical compromise between perceptual quality and computational cost.Source: §3.1.3 · p. 15, Table 3 · p. 16

3.2 Qualitative Evaluation

Qualitative results show SDM preserving some smaller structures and pathologies better than Pix2Pix, while the refiner mainly changes texture and contrast without grossly disrupting major anatomy in the examples.

3.2.1 Simple Scenarios

  • SDM better preserved smaller tissues near the liver, including pancreas, stomach, and tumor regions, which Pix2Pix often rendered as liver; it also maintained the gallbladder’s anechoic appearance more faithfully.Segmentator S reflected these differences in its predicted labels.Source: §3.2.1 · p. 16, Figure 4 · p. 17
  • SDM reproduced rib shadows in some cases, requiring 1000 steps for upper-boundary ribs, while Pix2Pix failed to reproduce ribs and shadows; Pix2Pix more closely matched the physical model’s elongated speckle texture.SDM’s speckle was more uniform and isotropic.Source: §3.2.1 · pp. 16–17

3.2.2 Pathological Scenarios

  • Both models rendered larger liver lesions and sagittal aortic enlargement, while SDM better retained smaller tumors, the axial aortic lumen, localized stone artifacts, and spatially defined polyps.Pix2Pix tended to blend small lesions or stones, distort or omit the axial aorta, and produce diffuse polyp effects; stones and polyps were introduced through label edits.Source: §3.2.2 · pp. 17–18

3.2.3 Stage II: Image Refinement (Qualitative)

  • SG-CycleGAN mainly changed texture, contrast, and speckle while preserving coarse anatomy in the representative cases, but SDM-derived outputs remained smoother and could lose fine texture or small boundaries.No gross hallucinated organs or complete removal of major structures were observed in these examples.Source: §3.2.3 · p. 18

4 Discussion

The discussion frames semantic-label synthesis as a way to approximate CT-based ray-casting without new patient-specific volumes at inference, while retaining CT-derived supervision for training. It highlights trade-offs between anatomical consistency, perceptual appearance, and refinement, and treats controllable pathology generation as proof of concept rather than clinical validation.

4 Discussion

  • Stage I approximates CT-based ray-casting from semantic labels at inference, but its training still requires anatomically plausible segmentations and ray-casting simulations.A relatively small number of CT volumes limits represented anatomical and pathological variability, and simulator or segmentation errors may propagate.Source: §4 · pp. 19, 22
  • SDM better preserves spatial and anatomical structure, while Pix2Pix more closely matches perceptual and distributional characteristics of the physics-based reference domain.The models are alternative operating points: SDM suits anatomy-focused generation, whereas Pix2Pix offers lower inference latency for latency-sensitive use.Source: §4 · pp. 20–21
  • SG-CycleGAN can refine either Stage I output toward real-US appearance, but Pix2Pix outputs are more compatible with its training distribution than smoother SDM outputs.Refinement modifies texture, speckle, scattering, and contrast, but cannot fully compensate for Stage I domain differences.Source: §4 · p. 21
  • Similar χ2 values after refinement suggest SG-CycleGAN can normalize regional intensity distributions across Pix2Pix and SDM inputs, although refinement may alter small findings or boundaries.No gross hallucination or removal of major structures was observed qualitatively; dedicated expert assessment is still needed to exclude subtle changes.Source: §4 · p. 21
  • Editing semantic labels produced qualitatively plausible appearances for liver tumors, abdominal aortic aneurysms, kidney stones, and gallbladder polyps, as proof-of-concept scenarios.The examples are not clinically validated disease models, and the study did not assess educational intervention or transfer to real scanning.Source: §4 · pp. 21–22
  • Semantic labels simplify tissue acoustics and cannot fully represent intra-organ heterogeneity, anisotropy, microstructure, vascularity, probe pressure, or patient-specific acquisition conditions.Pathological examples were qualitative only; correspondence with real pathological appearances has not been quantitatively or clinically validated.Source: §4 · p. 22
  • Stage II was trained on 11 healthy volunteers scanned with one scanner and protocol, so its evaluation cannot establish pathological realism or represent the full acquisition variability.Broader multi-center, multi-scanner, and pathology-rich data, plus external and expert validation, are identified as future needs.Source: §4 · p. 22
  • The study is a technical feasibility assessment, not evidence of clinical realism or educational effectiveness; expert assessment is needed to evaluate subtle refinement changes.The authors call for perceptual and distributional metrics alongside anatomy-aware measures, acoustic-artifact assessment, and blinded expert review.Source: §4 · pp. 22–23

Declarations

The declarations report author roles, funding, ethical status, conflicts, and resource availability.

Declarations

  • All authors contributed to interpretation, manuscript revision, and approval; Vitale and Deangeli developed and validated the models and performed experiments.Orlando coordinated the project, and Orlando and Larrabide jointly supervised it.Source: Declarations · p. 23
  • The project was funded by PIP GI 2021-2023-11220200102472CO and received a NVIDIA Quadro P6000 hardware donation.Source: Declarations · p. 23
  • Study datasets and processed data are publicly available, but complete source code cannot be released due to project-related restrictions.Reproducibility materials and instructions are available through the stated project resources.Source: Declarations · p. 23
  • The authors report no relevant financial or non-financial conflicts, and no human-participant or animal studies were performed.Informed consent was not applicable.Source: Declarations · p. 23
Loading 2610.08849v1…