PaperlayerView exact revision on arXiv

Source-linked AI summary

SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models

Zefan Yang, Zheyuan Zhang, Boris Maihe, Yikang Liu, Hang Yu, Ankush Mukherjee, Xiao Chen, Pingkun Yan, Terrence Chen, Shanhui Sun

arXiv:2610.19327v1cs.CV

TL;DR

Robotic ultrasound policy learning lacks realistic, controllable environments, and ultrasound’s unobserved 3D anatomy makes small probe motions produce large image changes. SonoWorld learns dynamics from unlabeled clinical videos, grounds them to metric 6-DoF actions with a small tracked subset, and distills the model for causal generation. In the reported controllability evaluation, the causal model remains close to its bidirectional teacher on pose-conditioned fidelity metrics.

  • Problem

    Robotic ultrasound lacks realistic, controllable environments for policy development, and existing ultrasound world models do not directly generate future observations from freehand 6-DoF probe actions.

    Source: §1 · p. 2

  • Method

    SonoWorld pretrains on unlabeled clinical ultrasound videos, grounds dynamics to metric probe actions using a small EM-tracked scan set, and distills the model for causal autoregressive generation.

    Source: §1 · pp. 2–3

  • Results

    The causal model’s pose-conditioned fidelity remains close to the bidirectional teacher, with a 0.75 dB decrease in PSNR, a 0.016 decrease in SSIM, and a 0.026 increase in LPIPS.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    SonoWorld demonstrates the potential of a data-driven interactive environment for robotic ultrasound data acquisition and policy evaluation.

    Source: §1 · p. 3, §6 · p. 10

    Action grounding uses a relatively small thyroid-only EM-tracked cohort, limiting validation to the action and anatomical distributions represented by those data.

    Source: §A.1 · p. 15

Abstract

from arXiv · show

Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an EM sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive generation with selectively caching eviction tokens at inference time. We evaluate controllability through pose-following fidelity, round-trip consistency, and axis-specific control under synthetic actions on held-out tracked scans, and further validate it with a user study. In model-based action discrimination, SonoWorld correctly ranks the target-directed probe motion over an equal-magnitude opposite motion in up to 74.8% of trials. To our knowledge, SonoWorld is the first causal, pose-controllable ultrasound world model learned primarily from freehand clinical videos, providing a data-driven interactive environment for robotic ultrasound data acquisition and policy evaluation.

1 INTRODUCTION

Robotic ultrasound needs realistic interactive environments for developing and evaluating probe-control policies, but existing approaches lack a reusable way to generate observations for counterfactual actions. SonoWorld learns ultrasound dynamics from unlabeled clinical videos, grounds them to 6-DoF probe actions using a small tracked subset, and converts the model for causal generation.

1 INTRODUCTION

  • Realistic interactive environments are needed to develop and evaluate robotic scanning policies, while real-patient trial-and-error is impractical and unsafe.Existing simulators trade clinical realism for controllability, and tracked-volume re-slicing cannot reproduce appearance changes under arbitrary probe orientations.Source: §1 · p. 2
  • Ultrasound world modeling is challenging because small probe motions can sharply change images of 2D cross-sections of unobserved 3D anatomy.Existing ultrasound world models use tracked acquisitions or latent-space prediction rather than directly generating future observations from freehand 6-DoF actions.Source: §1 · p. 2
  • SonoWorld learns dynamics from around 6.9 million unlabeled multi-organ frames and grounds them to physical probe control with a small set of EM-tracked thyroid scans.The tracked data are used for action grounding, not for large-scale dynamics pretraining.Source: §1 · pp. 2–3
  • SonoWorld distills the bidirectional model into a causal autoregressive generator using self-forcing and selective KV-cache retention for extended action-conditioned rollouts.The contribution describes cache retention as preserving generation quality during streaming inference.Source: §1 · p. 3
  • The evaluation includes measured, accelerated, round-trip, and axis-specific synthetic probe motions, alongside experienced-rater assessments of image quality, video consistency, and probe–anatomy alignment.The user study supplements the synthetic-action evaluations.Source: §1 · p. 3

2 RELATED WORK

Prior work spans controllable simulators, guidance models, and medical-video world models, each with different constraints on realism, tracking, or direct image generation. SonoWorld combines clinical-video dynamics learning with metric probe-action grounding and causal future-observation generation.

2 RELATED WORK

  • Latent-action world models separate learning visual dynamics from abundant unlabeled video from grounding those dynamics to physical actions, where paired action-video data are scarce.The related-work examples include Genie, AdaWorld, and DreamDojo.Source: §2 · p. 3
  • Medical-video world models include endoscopy and surgery, while ultrasound models have mainly supported representation learning or probe guidance rather than interactive image-generating environments.Fan et al. predict observations from tracked probe motions; SonoWorld instead learns primarily from untracked clinical videos and grounds actions with a tracked subset.Source: §2 · p. 3
  • Figure 2 summarizes SonoWorld’s progression from latent-action pretraining on unlabeled multi-organ videos, through thyroid pose grounding, to causal distillation.The final stage uses distribution matching distillation (DMD).Source: Figure 2 · p. 4
  • Physical phantoms and physics-based simulators offer pose exploration but lack clinical appearance, while real-patient scans are unsafe for trial-and-error and cover only the acquired trajectory.Re-sliced tracked volumes also freeze view-dependent effects and require a tracked acquisition for each subject.Source: §2 · pp. 3–4

3 METHODS

SonoWorld first learns ultrasound video dynamics from latent actions, then grounds the dynamics to metric probe motion, and finally adapts generation for causal interactive rollouts.

3.1 FOUNDATIONAL ULTRASOUND VIDEO DIFFUSION MODEL

  • The latent-action model compresses adjacent-frame dynamics into 32-dimensional actions and reconstructs the next frame from the current frame and its latent action.The encoder uses spatial and temporal attention, with a bottleneck intended to capture transition dynamics rather than shared appearance.Source: §3.1 · p. 4
  • The video diffusion model predicts future observations from an initial frame and latent-action sequence, injecting actions through adaptive layer normalization and training with flow matching.The clip is encoded with a pretrained Wan VAE; the action projection is zero-initialized to preserve the pretrained video prior at training start.Source: §3.1 · pp. 4–5

3.2 POSE-CONTROLLABLE VIDEO DIFFUSION MODEL

  • EM-tracked sensor poses are calibrated to the probe tip, then consecutive poses become relative 6-DoF actions that replace latent actions during post-training.The pose preprocessing corrects the sensor-to-tip lever arm so the resulting motion corresponds to the imaging plane.Source: §3.2 · p. 5, Figure 3 · p. 6
  • Relative poses cancel the patient frame and express translation in the starting probe’s local axes; translation and the rotation logarithm map each motion into R6.The resulting action is invariant to left-multiplying the pose stream by a fixed transform, with translation in millimetres and rotation in radians.Source: §3.2 · p. 5
  • Post-training represents cumulative motion within each four-frame window as a concatenated 24-dimensional action and conditions generation on a clean initial latent frame.The metric-action pathway and prefix convolution are zero-initialized, and the generator is trained end-to-end.Source: §3.2 · pp. 5–6

3.3 CAUSAL VIDEO DIFFUSION MODEL

  • The causal model uses backward-only block attention and self-forcing distillation to reduce training–inference context mismatch during autoregressive generation.It retains high-attention cached frames alongside an attention sink and recent window; inference reaches 40.6 fps at 256 × 256 resolution with four denoising steps on one NVIDIA H100.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate SonoWorld’s datasets, pose controllability, latent-action pretraining, causal training, and bounded-cache inference. Results show action-responsive generation, gains from latent actions and self-forcing, and strong quality with fewer cached frames.

4.1 DATASETS

  • The unlabeled clinical corpus contains 6,925,140 training frames, and the EM-tracked thyroid dataset comprises 42 subjects split into 33 training and 9 validation subjects.The clinical corpus spans abdomen, cardiac, and thyroid scans; the tracked scans use synchronized 6-DoF pose measurements.Source: §4.1 · p. 7

4.2 RESULTS ON ULTRASOUND WORLD MODEL CONTROLLABILITY

  • On EM-tracked scans, causal generation decreases 0.75 dB PSNR and 0.016 SSIM and increases 0.026 LPIPS versus the bidirectional teacher, while responding to commanded motion.Evaluations include pose replay, zero motion, doubled speed, and round trips; Figure 4 visualizes autoregressive futures from the same initial frame under different probe trajectories.Source: §4.2 · p. 7, Table 1 · p. 8, Figure 4 · p. 8

4.3 EFFECT OF PRETRAINING WITH LATENT ACTIONS

  • Figure 4 shows causal autoregressive future-frame generation conditioned on one initial ultrasound frame and different probe trajectories.Source: Figure 4 · p. 8
  • On an organ-balanced held-out set of 500 clips, with each rollout conditioned on the first three latent frames and generating the remaining nine, latent-action conditioning improves PSNR by 3.60 dB and SSIM by 0.083 while reducing LPIPS by 0.067.Source: §4.3 · p. 8

4.4 ABLATION STUDIES ON CAUSAL MODEL TRAINING METHODS

  • Self-forcing achieves the best perceptual quality among causal training variants, with SSIM 0.394 and LPIPS 0.436, while remaining within 0.15 dB PSNR of the bidirectional reference.Teacher forcing has higher PSNR at 16.44 dB but worse LPIPS at 0.515; the reported results favor training with the model’s generated context for long causal rollouts.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 ABLATION STUDIES ON CAUSAL INFERENCE TECHNIQUES

  • Selective eviction achieves the best PSNR (16.37) and SSIM (0.394) among tested cache configurations while matching the best LPIPS (0.436).All bounded-cache variants remain within 0.14 dB PSNR, 0.007 SSIM, and 0.004 LPIPS of full attention while retaining as few as 12 of 33 frames.Source: §4.5 · p. 9, Table 4 · p. 9

5 APPLICATIONS OF ULTRASOUND WORLD MODEL

The applications assess generated rollouts with expert ratings and test whether SonoWorld can discriminate target-directed from opposite probe motions. The model achieves up to 74.8% success in action discrimination, while raters score probe–anatomy alignment highest among the three criteria.

5.1 INTERACTIVE APPLICATION

  • In ratings of 55 generated clips, probe–anatomy alignment scores highest, followed by video consistency, while image quality scores relatively lower.Five experienced raters evaluated the clips on these three criteria using 5-point scales.Source: §5.1 · p. 10, Table 5 · p. 10

5.2 MODEL-BASED ACTION SELECTION

  • SonoWorld ranks the target-directed motion above an equal-magnitude opposite motion in 74.8% of trials at displacement threshold 15 and 66.7% at threshold 20.Success means the generated terminal observation for the target-directed candidate has higher PSNR to the target frame.Source: §5.2 · p. 10, Table 6 · p. 10

6 CONCLUSION

SonoWorld generates future ultrasound observations conditioned on metric 6-DoF probe motion, with thyroid action grounding and potential extension to other organs given tracked data. The authors identify unobserved tissue deformation and physiological motion, and the absence of a shared public benchmark, as limitations.

6 CONCLUSION

  • SonoWorld generates future observations conditioned on metric 6-DoF probe motion, with action grounding evaluated on thyroid scans and possible extension to other organs given appropriate tracked data.The model learns from around 6.9 million unlabeled clinical ultrasound frames and uses a small set of EM-tracked freehand scans for physical action grounding.Source: §6 · p. 10
  • Unobserved tissue deformation and physiological motion mean probe pose does not fully determine ultrasound appearance.Source: §6 · p. 11
  • The absence of a shared public benchmark prevents direct quantitative comparison with the closest action-conditioned ultrasound world model.Source: §6 · p. 11

A.1 LIMITATION OF SONOWORLD

SonoWorld’s action grounding is based on a small, thyroid-only EM-tracked cohort, limiting validation to the action and anatomical distributions represented in those data. Other unobserved factors and the lack of a shared benchmark further constrain interpretation and comparison.

A.1 LIMITATION OF SONOWORLD

  • The thyroid-only tracked cohort limits validation to the action and anatomical distributions represented in those data.Probe pose also leaves contact force, tissue deformation, physiological motion, and scanner settings unobserved.Source: §A.1 · p. 15
  • Pixel-level PSNR, SSIM, and LPIPS provide only indirect evidence of geometric control, and null-action rollouts exhibit an issue.The passage does not specify the nature of the null-action issue.Source: §A.1 · p. 15
  • Without a shared tracked-ultrasound benchmark, SonoWorld cannot be quantitatively compared directly with the closest action-conditioned ultrasound world models.The authors frame SonoWorld as a learned, data-driven environment rather than a physically guaranteed ultrasound simulator.Source: §A.1 · p. 15

A.2 DETAILED DERIVATION FOR POSE CALIBRATION AND RELATIVE POSE CONVERSION

This section derives tracked probe actions from calibrated sensor poses, accounting for the probe-tip offset and patient motion. It also describes the latent action model’s frame-transition code and training setup.

A.2 DETAILED DERIVATION FOR POSE CALIBRATION AND RELATIVE POSE CONVERSION

  • A 119.2 mm probe-to-sensor lever arm is composed onto each tracked sample to recover the probe-tip pose.The thyroid calibration includes a measured 1.7° tilt rather than an ideal CAD alignment.Source: §A.2 · p. 16
  • Patient-reference compensation is used only when mean sensor baseline is at most 300 mm; otherwise poses remain in the world frame.Of 42 retained thyroid segments, 35 are patient-referenced and 7 remain world-frame segments.Source: §A.2 · p. 16
  • Relative actions use probe-frame translation and rotation, making them invariant to fixed world-frame changes and expressing translation in the start pose’s local axes.The rotation is represented by a rotation vector, a continuous minimal three-dimensional parameterization without gimbal lock or double cover.Source: §A.2 · pp. 16–17
  • Each pixel frame’s action records cumulative motion from its four-frame window’s start, preserving within-window trajectory rather than only net displacement.Four pixel-frame motions are grouped per latent frame, with a zero action prepended for the first latent frame.Source: §A.2 · p. 17

A.3 LATENT ACTION MODEL TRAINING

  • The latent action model infers a compact transition code from adjacent frames, and its decoder must use that code to reconstruct the target from source-frame tokens.The code is thereby encouraged to represent frame changes, including probe motion and resulting anatomical displacement, rather than anatomy itself.Source: §A.3 · pp. 17–18, Figure 5 · p. 18
  • Training samples target frames one to four frames after the source and combines pixel reconstruction with a KL regularizer toward a standard normal prior.At inference, the posterior mean is used; the small β makes the low-dimensional code and decoder’s lack of target-frame access the main bottleneck.Source: §A.3 · p. 18

A.4 IMPLEMENTATION DETAILS

The implementation scales pretraining through cached data and a staged curriculum, then adapts the model to scarce tracked scans with online pose-consistent augmentation. Its objective combines generated-frame prediction with motion consistency.

A.4 IMPLEMENTATION DETAILS

  • Pretraining uses a 153,892-clip multi-organ corpus, with cached latents, actions, and text embeddings enabling an effective batch of 768 clips.The corpus is 69.5% abdomen, 17.0% cardiac, and 13.5% thyroid clips.Source: §A.4 · p. 18
  • Post-training uses 42 tracked thyroid scans yielding 262 clips, with online temporal augmentations applied consistently to frames and poses before actions are recomputed.The four augmentation cases are sampled with equal probability; stop-motion repeats frames to place exact zero actions in training.Source: §A.4 · p. 19, Table 8 · p. 19
  • Each 45-frame post-training clip maps its 44 per-frame 6-DoF motions into 11 concatenated 24-dimensional chunk actions, plus a zero action for the first latent frame.The pretrained action embedding is replaced with a newly initialized 24-dimensional embedding while other parameters are inherited.Source: §A.4 · pp. 19–20
  • The objective adds motion consistency because framewise prediction can render plausible frames while ignoring actions, and compares predicted with target frame-to-frame changes.The motion term is masked to transitions ending in generated frames, including the prefix-to-generation handoff.Source: §A.4 · p. 20
  • A four-stage curriculum progresses from single latent frames to video clips, prefix conditioning, and finally latent-action conditioning.Each stage warm-starts the next and trains the full network.Source: §A.4 · p. 20, Table 9 · p. 20

B.1 BENCHMARKING VIDEO VAES FOR ULTRASOUND VIDEO RECONSTRUCTION

The benchmark finds similar reconstruction quality across four video autoencoders, while memory and throughput distinguish them. Wan2.1-T2V-1.3B is selected for its efficiency and competitive reconstruction metrics.

B.1 BENCHMARKING VIDEO VAES FOR ULTRASOUND VIDEO RECONSTRUCTION

  • Wan2.1-T2V-1.3B matches the best LPIPS at 0.033 and trails the best PSNR by 0.12 dB, while offering the lowest peak memory and highest throughput.Its SSIM is the lowest of the four by 0.022.Source: §B.1 · p. 21
  • Wan2.1 reconstruction reaches 36.50 dB PSNR and 0.033 LPIPS, compared with 16.09 dB and 0.439 for causal generation in Table 1.The authors conclude that the autoencoder is not the bottleneck on generation quality.Source: §B.1 · p. 21
  • 36.24–36.62 dB PSNR and 0.900–0.922 SSIM span the four autoencoders, while HunyuanVideo-1.5 uses 18.85 GB peak memory versus 2.22 GB for Wan2.1.HunyuanVideo-1.5 runs at half the throughput and has the worst LPIPS by a factor of 1.9.Source: Table 10 · p. 21, §B.1 · p. 21

B.2 RESULTS ON PRETRAINING WITH LATENT ACTIONS

Latent actions carry transition information beyond the source frame and improve multi-organ video generation. Predictions recover anatomy and displacement but smooth unpredictable speckle.

B.2 RESULTS ON PRETRAINING WITH LATENT ACTIONS

  • Across thyroid, cardiac, and abdominal scans, latent-action predictions recover anatomy and displacement but smooth speckle that decorrelates between frames.Difference maps highlight the same regions across organs, indicating transition-relevant information beyond the source frame's appearance.Source: §B.2 · p. 21
  • 4.74 dB PSNR, 0.114 SSIM, and 0.121 LPIPS are the average gains from latent actions across organs in 8-second generation.Gains were largest on abdominal scans (+6.42 dB) and smallest on thyroid scans (+3.21 dB); thyroid remained the hardest organ in both settings.Source: §B.2 · p. 21
  • With latent actions, generated sequences track the ground-truth video, while models without them drift into plausible but unrelated continuations.Because the action branch is zero-initialized, the comparison measures the predictive value of transition information beyond visual context.Source: §B.2 · p. 21

B.3 ABLATION STUDIES: EFFECT OF LATENT-ACTION PRETRAINING ON POSE-CONDITIONED MODELS

Latent-action pretraining improves pose-conditioned generation across probe trajectories, and causal rollouts extend beyond the training horizon. The user-study interface lets raters assess image quality, video consistency, and probe–anatomy alignment.

B.3 ABLATION STUDIES: EFFECT OF LATENT-ACTION PRETRAINING ON POSE-CONDITIONED MODELS

  • 0.46 dB PSNR is the round-trip advantage from latent-action pretraining, the largest reported across probe trajectories.Pretraining improves every metric under every trajectory; gains are smallest under zero motion, and the advantage persists in 8-second rollouts beyond the 3-second training horizon.Source: §B.3 · p. 22

B.4 RESULTS ON CAUSAL INFERENCE

  • At t = 8 s, causal rollouts on three held-out scans follow the ground-truth anatomy despite extending inference to 33 latent frames from 12-frame training clips.The model generates autoregressively in four denoising steps from the initial frame and measured 6-DoF pose sequence.Source: §B.4 · p. 22, Figure 8 · p. 24
  • With latent actions, generated frames follow ground-truth anatomy as the probe moves across thyroid, cardiac, and abdominal scans.The model receives the first nine frames and the latent-action sequence, then generates the remaining frames.Source: Figure 7 · p. 23

C USER STUDY INTERFACE

  • Raters score image quality, video consistency, and probe–anatomy alignment on 5-point scales while viewing generated clips with thyroid segmentation overlays.The interface also shows the segmented thyroid in 3D alongside the probe path, probe frame, and current imaging plane.Source: C USER STUDY INTERFACE · p. 23, Figure 10 · p. 25

Related papers

Loading 2610.19327v1…