PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Relevant Feature-Dynamics Fidelity Enables Zero-Shot Sim-to-Real Transfer for Robotic Ultrasound Scanning

Yizhao Qian, Jiayuan Luo, Wanyi Zhu, Yameng Zhang, Max Q. -H. Meng, Yixuan Yuan, Li Liu

arXiv:2608.29516v1cs.RO

TL;DR

Raw-B-mode robotic ultrasound policies need substantial interaction data, but real-robot collection is costly and safety-constrained, making reliable zero-shot sim-to-real transfer an open challenge. The paper formalizes TR-FDF, develops a simulator around it, and reports 390 successful deployments out of 400 while showing that TR-FDF sensitivity complements single-frame realism.

  • Problem

    Reliable strict zero-shot deployment of policies trained exclusively on simulated raw B-mode observations remains unestablished, despite costly and safety-constrained real-robot data collection.

  • Method

    The paper develops an ultrasound observation generator combining a shared structural intermediate domain, trajectory-level fixed-X0 rendering, and few-step flow generation.

  • Results

    390 of 400 zero-shot real-robot deployments succeeded, while ablations showed that comparable FID could accompany sharply worse TR-FDF sensitivity and transfer.

  • Takeaways & Limitations

    TR-FDF sensitivity should be evaluated jointly with single-frame realism and rollout efficiency when assessing observation generators for zero-shot transfer.

  • Takeaways & Limitations

    TR-FDF sensitivity may remain correlated with FID, and the proxy, generator, and validation were developed specifically for robotic ultrasound.

Abstract

from arXiv · show

Robotic ultrasound policies operating directly on B-mode images require extensive interaction data, whereas real-robot data collection is costly and safety-constrained. Simulation provides a scalable alternative, but zero-shot transfer depends not only on single-frame realism but also on whether simulated observations reproduce task-relevant feature changes induced by probe motion. We term this cross-domain consistency task-relevant feature-dynamics fidelity (TR-FDF). Under local regularity assumptions, our contraction analysis shows that greater sensitivity of TR-FDF mismatch to probe motion reduces the effective closed-loop contraction margin, whereas motion-independent errors primarily enlarge the residual error bound. Guided by this analysis, we develop a TR-FDF-oriented ultrasound simulator that combines a shared structural intermediate domain, trajectory-level fixed noise, and few-step conditional flow generation. In phantom experiments, a policy trained exclusively in simulation succeeded in 390 of 400 zero-shot deployments across four target planes. The simulator achieved an FID of 29.66 and generated observations at 67.1 Hz. Controlled interventions, ablations, and baseline comparisons showed that TR-FDF sensitivity complements single-frame realism in predicting zero-shot transfer performance.

I. INTRODUCTION

The paper asks what an ultrasound observation generator must preserve for raw-B-mode policies to transfer zero-shot, arguing that motion-induced task-relevant feature changes matter alongside single-frame realism. It develops a simulator targeting these dynamics and evaluates transfer on a real phantom platform.

  • Motivation: Real-robot reinforcement learning is costly and safety-constrained, motivating simulation for learning image-to-action relationships.Raw B-mode policies typically require many interaction episodes.
  • Research gap: Zero-shot transfer requires simulated observations to reproduce task-relevant feature changes induced by the same probe motion as in the real system.Single-frame appearance similarity alone does not ensure a consistent observation-action relationship.
  • Analysis: TR-FDF mismatch sensitivity to probe motion reduces the real closed-loop contraction margin, whereas motion-independent errors mainly enlarge the residual-error region.This analysis motivates evaluating dynamics rather than relying only on static image quality.
  • Method: The proposed generator combines a shared structural intermediate domain, trajectory-level fixed-X0 rendering, and few-step flow generation.These components target pose-driven feature evolution, single-frame realism, and rollout efficiency.
  • Evaluation: The study evaluates TR-FDF through controlled interventions, component ablations, and comparisons with external baselines rather than simulator-side success alone.The paper treats TR-FDF as complementary to single-frame realism for predicting real closed-loop transfer.

III. PROBLEM FORMULATION AND ANALYSIS

This section formalizes how probe motion, task features, and observations evolve across simulated and real domains, then defines TR-FDF mismatch as a distributional comparison of feature transitions. It distinguishes dynamic consistency from single-frame realism for closed-loop transfer.

  • III. PROBLEM FORMULATION AND ANALYSIS: The analysis asks which observation discrepancies affect transfer, how they alter closed-loop error and stability margins, and what an observation generator should satisfy.These questions organize the problem formulation and analysis.
  • A. Task Setup and Notation: A probe motion changes the current B-mode observation and influences subsequent states through the next policy action.The closed-loop sequence links probe pose, image-derived features, actions, and next states.
  • A. Task Setup and Notation: The notation distinguishes simulated and real domains while associating each domain with its preceding probe motion and task-relevant feature.The feature is extracted from the domain-specific B-mode image.
  • A. Task Setup and Notation: The policy outputs an action from the current observation, driving the system to the next state and making motion-consistent observation changes central to transfer suitability.Single-frame realism is therefore insufficient by itself.
  • A. Task Setup and Notation: For a one-step probe motion, the formulation compares the conditional distributions of task-feature increments across simulation and reality.The comparison concerns feature transitions under the same motion and fixed local task conditions.
  • A. Task Setup and Notation: A smaller DTR indicates higher TR-FDF because DTR measures whether identical probe motions produce consistent feature transitions across domains.FID instead measures similarity between single-frame observation distributions.

B. How Simulator Bias Enters Transfer: A Local Perturbation Analysis

The analysis separates motion-dependent TR-FDF mismatch from motion-independent residual errors in real closed-loop transfer. Under local regularity assumptions, motion sensitivity reduces the contraction margin, while stationary mismatch and residual bias enlarge the ultimate error neighborhood.

  • Local assumptions: Local Lipschitz and finite-gain assumptions allow sim–real feature-transition mismatch to propagate through the policy, dynamics, and task-error chain.The analysis compares simulated and real transition kernels from the same current state near the target.
  • Transfer condition: ρreal < 1 preserves local mean-square contraction into a residual neighborhood, whereas ρreal ≥ 1 provides no convergence guarantee from simulator convergence alone.The residual neighborhood is determined by β.
  • Motion-dependent mismatch: LTR enters the effective contraction factor, so greater sensitivity of TR-FDF mismatch to probe motion directly reduces the contraction margin.LTR measures how rapidly feature-transition mismatch grows with probe-motion magnitude.
  • Motion-independent mismatch: d0 and B combine as β = Cd0 + B and mainly enlarge residual error near the target rather than directly changing the contraction factor.d0 captures stationary-probe mismatch, while B aggregates static task-feature bias, physical discrepancies, and other residual effects.

C. Design Criteria for the Observation Generator

The observation generator must jointly preserve task-relevant feature responses to probe motion, control motion-independent observation error, and meet rollout-speed requirements. These criteria distinguish contraction-margin effects from residual-error effects and impose an engineering constraint for policy training.

  • Criterion 1: TR-FDF: LTR directly affects the real closed-loop contraction factor by measuring sensitivity of sim–real feature-transition mismatch to probe-motion magnitude.The generator must preserve correct task-feature responses, not merely visually smooth changes.
  • Criterion 2: Motion-Independent Observation Quality: β = Cd0 + B captures motion-independent residual effects, requiring stable observations under identical imaging conditions and a close single-frame generation distribution.d0 represents stationary-probe feature jitter, while the generator-related part of B represents static task-feature bias.
  • Criterion 3: Rollout Efficiency: The generator must satisfy the rollout time budget required for step-wise closed-loop policy training.This criterion determines whether the generator can support large-scale policy training.
  • Combined requirements: The proposed design addresses the criteria through shared structural representation, fixed-X0 trajectory rendering, conditional B-mode generation, and few-step flow inference.Together, these components target motion-related mismatch, additive observation error, and inference speed.

IV. METHODOLOGY

The methodology combines a three-stage observation generator with fixed-X0 trajectory rendering to align structural responses, realistic B-mode appearance, and rollout efficiency. Stage I and structure-conditioned Stage II primarily reduce motion-related TR-FDF mismatch, while the remaining components control additive error and speed generation.

  • Motion-related fidelity: The generator reduces LTR through a shared structural domain and structure-conditioned rendering that preserve pose-driven anatomical changes.Stage I establishes the structural representation, while Stage II preserves its changes in generated B-mode images.
  • Motion-independent quality: Fixed-X0 rendering suppresses frame-wise sampling jitter and primarily reduces d0 during trajectory generation.This targets motion-independent feature variation across observations.
  • Pipeline: Probe poses are converted to structural masks, encoded into a shared latent space, rendered as B-mode observations, and accelerated through the three-stage pipeline.Fixed-X0 trajectory rendering suppresses motion-independent sampling variation.
  • Motion-independent quality: Stage II generates realistic B-mode appearance and reduces the generator-induced component of static observation bias in B.Together with fixed-X0 rendering, it controls the additive observation error Cd0 + B.
  • Rollout efficiency: Stage III uses reflow distillation to compress a high-step flow teacher into a low-step student that meets the rollout time budget.The compression aims to preserve structural responses and single-frame quality as far as possible.

2) Network Architecture:

The architecture uses Stage I to learn a shared structural latent that preserves pose-dependent anatomy while suppressing modality-private appearance, then uses Stage II to render B-mode images conditioned on that structure. Its objectives preserve TR-FDF dynamics, prevent representation collapse, and maintain a bounded response to structural changes.

  • Stage I architecture: Stage I uses mask and ultrasound encoders plus a segmentation decoder to map both modalities into a shared structural latent domain.The latent is designed to preserve spatial anatomy while suppressing speckle, brightness, and device-style factors.
  • Stage-I objectives: Ldist preserves pose-induced latent geometry, while Lsmooth enforces local continuity related to motion-dependent response magnitude.These objectives target the latent dynamics associated with LTR.
  • Stage-I objectives: Lseg, Lalign, and Lreg preserve anatomical information, align mask and ultrasound representations, and stabilize latent scale against degenerate solutions.Together they establish a shared, non-collapsed structural space.
  • Training data: Stage I is pretrained on low-quality paired data and a small annotated dataset, using the paired data to learn structure rather than final realistic appearance.The paired data have FID 96.81 and differ from real ultrasound in sharpness, acoustic-window shape, and structural projection relationships.
  • Stage II renderer: Stage II renders realistic ultrasound appearance over the bounded structural trajectory produced by Stage I while remaining conditioned on that trajectory.The CFM renderer maps initial noise to B-mode images through a latent-conditioned velocity field, so Stages I and II jointly reduce LTR.

3) Real-Ultrasound Consistency Constraint:

The simulator learns real-ultrasound appearance from unpaired images and improves rollout efficiency through fixed trajectory-level noise and reflow distillation. Its evaluation examines whether TR-FDF is relevant to zero-shot transfer and how simulator components support fidelity and efficiency.

  • Real-ultrasound consistency: Stage II learns real-ultrasound appearance from unpaired images by reconstructing each image from its own frozen structural latent.This avoids requiring paired simulated masks and real ultrasound images.
  • Rollout efficiency: Stage III compresses a high-NFE teacher into a low-step student through reflow distillation for faster closed-loop rollouts.The teacher provides better image quality, but its inference speed is insufficient for large-scale policy rollouts.
  • Rollout efficiency: NFE=4 enables B-mode observation generation at rollout speed while aiming to preserve the teacher’s appearance statistics and structure-conditioned responses.Stage III targets inference speed rather than further improving TR-FDF.
  • Fixed-X0 trajectory rendering: Independent frame sampling creates pose-independent inter-frame jitter, separating structure-driven observation dynamics from generator-side resampling noise.The structure-driven term depends on the latent change, whereas the resampling term is independent of probe pose.
  • Fixed-X0 trajectory rendering: Reusing one sampled noise value per trajectory removes generator-side resampling jitter without changing network weights or marginal single-frame noise.This targets the additive-error route by reducing the generator-induced contribution to d0 without directly changing the contraction factor ρreal.
  • Evaluation scope: The experiments test multi-plane zero-shot transfer, TR-FDF sensitivity, component contributions, external baselines, and robustness to contact-force and target-motion disturbances.The evaluation focuses on whether TR-FDF is a measurable observation-model property relevant to raw-B-mode transfer.

A. Experimental Platform, Policy, and Evaluation Protocol

The study evaluates a raw-B-mode policy on a real robotic ultrasound platform using a custom abdominal phantom, four target planes, and standardized zero-shot success criteria. The proposed simulator produced fast, high-quality observations, and simulation-only training yielded high real-robot success across 400 deployments.

  • Experimental platform: The platform combines a UR5 manipulator, Mindray M9 scanner, convex probe, force sensor, and custom abdominal phantom containing multiple abdominal structures.The phantom provides multiple anatomical structures and target planes for real-robot evaluation.
  • Policy: The policy uses a ResNet-50 visual front end, MLP control head, and SAC, with 5DoF image-based motion control plus low-level force control.The architecture was intentionally conventional rather than highly task-specialized.
  • Evaluation protocol: 100 independent episodes were used per target or condition, with success defined by final position error below 3 mm and rotation error below 5°.Four target planes therefore produced 400 real-robot episodes in the main transfer test.
  • TR-FDF measurement: The TR-FDF sensitivity proxy estimates the maximum local slope of feature-transition mismatch across motion bins, while FID is reported separately for single-frame quality.The proxy is an empirical estimate under finite motion bins and samples, not the theoretical LTR upper bound.
  • Simulator characteristics: 31.61 TR-FDF sensitivity, 29.66 FID, and 67.1 Hz inference characterized the four-step simulator used for policy training.The training observations were 224×224 grayscale B-mode images targeting hepatic-vein, kidney, aorta, and liver-tip regions.
  • Zero-shot transfer: 390/400 real-robot episodes succeeded, giving a 97.5% overall zero-shot success rate across four target planes.A2 and A3 achieved 100/100, A1 achieved 98/100, and A4 achieved 92/100; all failures were timeouts.

D. Controlled Interventions on TR-FDF Sensitivity

Controlled lookup-table and learned-simulator interventions varied motion-dependent observation mismatch while keeping single-frame image quality comparable. In both settings, higher TR-FDF sensitivity coincided with sharply lower real-world success and larger simulator-to-real gaps.

  • Real-image lookup intervention: The real-image lookup intervention perturbed the pose-to-observation relationship while controlling appearance realism and temporal noise flicker.A fixed displacement field and an unperturbed target isolated motion-dependent mismatch away from the target.
  • Real-image lookup intervention: FID stayed within 0.628–1.100 while TR-FDF sensitivity rose from 10.99 to 54.82 and real-world success fell from 100% to nearly 0%.The result supports a negative relationship between TR-FDF sensitivity and real-world success under comparable image quality.
  • Real-image lookup intervention: At d=5.0 mm, simulator success remained 100/100 while real-world success fell to 69/100 at TR-FDF sensitivity 18.09.At d=10.0 mm and 12.5 mm, simulator success was 78/100 and 77/100, whereas real-world success was 11/100 and 3/100.
  • Experimental scope: The lookup-table experiment was limited to two dimensions because a 6DoF table would require an impractically large real-image dataset.The learned-simulator experiment supplied the complementary 6DoF perturbation test.
  • Learned-simulator structural intervention: The learned-simulator structural intervention preserved visible anatomy and comparable single-frame fidelity while changing how neighboring structures evolved with pose.This tested the same failure mode in the complete learned-simulator and 6DoF pipeline.
  • Learned-simulator structural intervention: TR-FDF sensitivity increased from 31.61 to 302.64 while real-world success decreased from 98/100 to 0/100, with perturbed FID remaining near 29.66.Simulator success over the final 100 episodes remained 99/100–100/100.
  • Cross-intervention conclusion: Across both interventions, higher TR-FDF sensitivity was associated with lower real-world success and a larger simulator-to-real success gap independent of FID.The experiments support TR-FDF sensitivity as a diagnostic complementary to single-frame image quality.

E. Ablation Studies

The component-level ablations test how simulator design choices affect TR-FDF sensitivity, single-frame realism, rollout efficiency, and real-robot transfer. Removing key components substantially degraded transfer or efficiency despite unchanged simulator-side success.

  • Component-Level Ablations: 55/100 real-robot success followed frame-level random X0, alongside higher TR-FDF sensitivity and FID despite nearly unchanged inference speed.TR-FDF sensitivity rose from 31.61 to 50.78, while FID rose from 29.66 to 43.36.
  • Component-Level Ablations: 5/100 real-robot success followed removal of the shared structural domain, while TR-FDF sensitivity reached 244.26 despite FID remaining near baseline.FID was 31.14 versus the baseline 29.66, indicating that comparable single-frame quality did not preserve feature dynamics or transfer.
  • Component-Level Ablations: 17/100 real-robot success followed weakened Stage-I shaping, with TR-FDF sensitivity increasing to 164.61 and FID to 53.05.The results support Stage-I shaping and the shared structural domain as important for pose-driven structural dynamics and cross-domain alignment.
  • Component-Level Ablations: 130.7 Hz inference after replacing CFM with a GAN came with FID of 210.83 and 18/100 real-robot success.The faster generator degraded single-frame realism and transfer performance.
  • Component-Level Ablations: 3.07 Hz inference without reflow distillation preserved 99/100 real-robot success, TR-FDF sensitivity of 30.43, and FID of 27.83, but hindered efficient rollouts.Reflow distillation primarily improved observation-generation speed rather than TR-FDF sensitivity or single-frame realism.
  • Component-Level Ablations: 100/100 simulator-side success was achieved by all five ablations, yet real-world performance and rollout efficiency differed substantially.Simulator-side learnability alone did not determine transferability.

2) Stage-I Loss-Level Ablations:

The loss-level and external-baseline studies separate the mechanisms supporting feature dynamics from those supporting realism and complete transfer. They show that local continuity, shared structural representation, and joint screening across metrics are important for reliable zero-shot deployment.

  • Stage-I Loss-Level Ablations: Removing Lsmooth increased TR-FDF sensitivity to 138.34, the largest change among the six loss ablations, highlighting the importance of local continuity.Removing Ldist changed sensitivity from 31.61 to 35.01 with nearly unchanged FID, whereas removing Linvar raised sensitivity and FID to 49.18 and 40.97.
  • Stage-I Loss-Level Ablations: Removing Lseg, Lalign, and Lreg raised TR-FDF sensitivity to 46.88, 35.43, and 49.96, with corresponding FID values of 63.36, 39.12, and 53.63.These losses support structural supervision, cross-modal latent alignment, and latent-scale regularization in the shared structural representation.
  • Stage-I Loss-Level Ablations: The loss-level study used offline metrics and did not perform complete real-robot evaluations for every variant, so it made no separate claims about individual-loss effects on real-robot success.The system-level role of Stage I was assessed through the component-level deployment study.
  • External Baselines: 100/100 simulator-side SAC success did not guarantee transfer: CycleGAN, CUT, and UNSB achieved it but only 1–7/100 real-robot success.Observation generators were therefore screened sequentially for generation, simulator policy learning, and real-robot closed-loop convergence.
  • External Baselines: UVCGAN2 achieved FID 21.81 and simulator success 92/100 but TR-FDF sensitivity 904.16 and zero real-robot successes, while CUT had sensitivity 42.61 but only 7/100 real-robot success.FID and TR-FDF sensitivity diagnosed different observation-model properties.
  • External Baselines: Only the proposed simulator jointly achieved TR-FDF sensitivity 31.61, FID 29.66, rollout speed 67.1 FPS, simulator success 100/100, and real-robot success 98/100.It maintained capability across all three sequential screening levels.

G. Task-Level Simulator Consistency under Disturbances

The simulator-trained policy retained zero-shot closed-loop behavior under contact-force variation and sustained phantom motion, supporting task-level consistency beyond static image realism. Controlled evidence further links TR-FDF sensitivity to real-robot transfer outcomes and motivates joint evaluation with realism and efficiency.

  • Contact-force variation: 94.0% and 98.0% success were achieved at 5 N and 15 N contact force, respectively, under zero-shot real-robot deployment.All other training and evaluation settings were held fixed.
  • External baselines: The proposed method’s trajectories converged around the target, whereas external baselines mainly produced off-target failure trajectories.The figure caption identifies green and red as successful and failed episodes, respectively, with a common target marker.
  • Sustained phantom motion: The actual probe trajectory followed both spiral and square target motions, including the square trajectory’s piecewise direction changes.These tests used a fixed simulator-trained policy without real-world fine-tuning or online adaptation.
  • Transfer consistency: Increasing TR-FDF sensitivity was accompanied by lower real-robot success and a separation between simulator-side success and real-world deployment outcomes, with FID largely unchanged.This result supports evaluating observation generators using TR-FDF sensitivity jointly with single-frame realism and rollout efficiency.
  • Scope: The local analysis applies in principle to action-conditioned observation modalities under corresponding local boundedness conditions, but empirical applicability beyond robotic ultrasound remains unestablished.The controlled intervention evidence and validation in this study are specific to ultrasound.
Loading 2608.29516v1…