PaperlayerView exact revision on arXiv

Source-linked AI summary

From Freehand Demonstrations to Autonomous Robotic Ultrasound: An Anatomy-Anchored Learning Framework

Yuan Bi, Fangxu Han, Ye Su, Yiping Zhou, Zhongliang Jiang, Nassir Navab

arXiv:2610.32525v1cs.RO

TL;DR

Robotic ultrasound imitation learning needs observation and pose representations suited to anatomy-guided scanning and probe–patient interaction. This paper combines landmark-based placement, anchor-relative multimodal navigation, and robot-free demonstrations; it reports 88–96% task success across four tasks and gains over a fine-tuned π0 model on abdominal tasks.

  • Problem

    Robotic-ultrasound imitation-learning systems use varied observation configurations, and the task-specific necessity and role of their modalities had not been systematically examined.

    Source: §I · p. 2

  • Method

    The framework combines anatomy-guided probe placement with anchor-relative Streaming Flow Policy navigation conditioned on ultrasound, wrist-camera views, contact force, and probe poses, using a handheld platform for robot-free demonstrations.

    Source: §I · p. 2

  • Results

    Across four tasks on unseen participants, success was 88–96%; versus fine-tuned π0, abdominal task success improved by 10–42 percentage points.

    Source: §I · pp. 2–3, §V · p. 17

  • Takeaways & Limitations

    The policies transferred without retraining to a second robot, achieving 83.3–100% success across all four tasks.

    Source: §V · p. 17

Abstract

from arXiv · show

Ultrasound imaging is highly operator-dependent, motivating robotic systems that reproduce expert scanning skills consistently. These skills involve coordinated interpretation of anatomical landmarks, ultrasound images, and probe-tissue interactions that are difficult to encode in hand-designed control strategies. Imitation learning offers an alternative, but its observation and action representations should reflect the anatomically structured nature of scanning. We propose an anatomy-anchored imitation-learning framework that combines anatomy-guided probe placement with learned ultrasound navigation. The placement module identifies a task-specific anatomical starting point from external visual observations and establishes the initial probe pose as the reference frame for scanning. A flow-based policy then generates scanning motions from anchor-relative probe poses, ultrasound images, probe-mounted camera views, and contact forces, while a termination head predicts task completion. This avoids continuous reliance on third-person vision and reduces dependence on the robot-patient configuration. A handheld acquisition platform enables multimodal demonstrations to be collected during natural freehand scanning without a robot. On 16 unseen participants, task success reaches 96%, 94%, 88%, and 94% for kidney, gallbladder, and liver standard-plane acquisition and radial artery sweeping, respectively. Compared with a fine-tuned pi_0 model, success on the abdominal tasks improves by 10-42 percentage points, with lower time-to-target, peak contact force, and trajectory jerk. Ablations support the complementary roles of anatomical anchoring, egocentric vision, and force feedback. End-to-end trials with automatic placement achieve 90-100% success, and the policies transfer without retraining to a second robot with 83-100% success.

I. INTRODUCTION

The paper frames robotic ultrasound as a task-specific imitation-learning problem whose observations and pose representation should reflect anatomy and probe–patient interaction. It combines anatomy-guided initialization, learned multimodal navigation, and robot-free demonstration collection, reporting high success across four tasks.

I. INTRODUCTION

  • Prior ultrasound imitation-learning systems vary in their observation inputs, but the task-specific necessity and role of these modalities had not been systematically examined.Source: §I · p. 2
  • Ultrasound images alone incompletely represent probe motion because they show 2D cross-sections and cannot reliably track out-of-plane motion.Source: §I · p. 2
  • Anchor-relative probe motion reduces dependence on robot placement, patient positioning, and workspace configuration while preserving scan progression.Source: §I · p. 2
  • Anatomy-guided placement establishes a task-specific starting pose, after which a learned policy navigates relative to the anchor using multimodal feedback.A termination head predicts when the target view has been reached.Source: §I · p. 2
  • A handheld platform synchronously records ultrasound, wrist-camera views, probe pose, and contact force during natural freehand demonstrations without a robot.Source: §I · p. 2
  • 88–96% task success across four tasks, with 10–42 percentage-point gains over fine-tuned π0 on abdominal tasks.The evaluation covers previously unseen participants.Source: §I · pp. 2–3

II. RELATED WORK

Imitation-learning research has progressed from sequence models to generative and streaming policies for multimodal, temporally coordinated actions. Handheld interfaces also decouple demonstration collection from robot availability, with ultrasound offering a way to capture demonstrations using the instrument used at deployment.

A. Imitation Learning in General Domain

  • Diffusion policies model multimodal actions through iterative denoising, while flow-based approaches aim to reduce inference overhead,.The related-work discussion also describes adaptive integration steps and single-step generation.Source: A. Imitation Learning in General Domain · p. 3
  • Streaming Flow Policy generates and executes action trajectories incrementally, supporting responsive closed-loop control without waiting for a full action chunk [8].Source: A. Imitation Learning in General Domain · p. 3

B. Handheld Demonstration Collection Platform

  • Portable interfaces collect demonstrations without the target robot; UMI transfers handheld-gripper demonstrations using a relative-trajectory representation [23].Source: B. Handheld Demonstration Collection… · p. 3
  • Ultrasound demonstrations can be recorded by manipulating the same instrument used in robotic execution, reducing the human-to-robot embodiment gap.The probe records synchronized ultrasound, local visual observations, contact force, and motion.Source: B. Handheld Demonstration Collection… · p. 4

C. Robot Learning for Robotic Ultrasound

Prior robotic-ultrasound learning spans movement primitives, image-driven policies, pose-conditioned policies, and multimodal methods, with differences in data collection and coordinate representations. The paper motivates a combined approach using anatomy-grounded observations, smooth trajectory generation, and portable acquisition.

C. Robot Learning for Robotic Ultrasound

  • Prior ultrasound-learning methods use varied observations and pose frames, while robot-base representations can depend on workspace configuration and anatomical frames may require reconstruction or calibration.Earlier approaches include image-only, pose-conditioned, and multimodal policies; some use robot teleoperation or physical guidance.Source: C. Robot Learning for Robotic Ultrasound · p. 4

III. METHODS

  • The method separates landmark-based probe placement from ultrasound-guided navigation, combining anchor-relative motion with ultrasound, wrist-camera, and force feedback.The policy uses a Streaming Flow formulation for smooth, temporally consistent probe motion.Source: §III · p. 5

A. Data Acquisition Platform

  • A portable acquisition platform is designed for natural freehand capture, sufficient multimodal sensing, and transfer to different robotic embodiments.The framework records demonstrations without requiring a robot in the data-collection loop.Source: A. Data Acquisition Platform · p. 5, §III · p. 5

1) Sensors:

The acquisition platform records synchronized probe pose, force, wrist-camera views, and ultrasound images, with calibration and temporal alignment supporting multimodal demonstrations. The policy uses anchor-relative probe states and a continuous velocity-field formulation for navigation.

1) Sensors:

  • The handheld device records probe pose, applied force, wrist-camera views, and ultrasound images; its wrist camera observes the probe footprint and local patient surface.A scene camera supplies broader patient and surface context for anatomical localization and initial placement.Source: 1) Sensors · pp. 5–6

2) Calibration and Synchronization:

  • Marker-to-probe calibration combines hand-eye calibration with the fixed CAD-derived end-effector-to-probe transform; streams are aligned to ultrasound timestamps, with short tracking gaps interpolated.Force calibration compensates for the weight of the probe and mounted components.Source: 1) Sensors · p. 6, 2) Calibration and Synchronization · p. 6

B. Anatomy-anchored Streaming Flow Policy

  • The Streaming Flow Policy conditions on observation history and the current probe pose, predicts a velocity field, integrates desired poses, and re-queries during receding-horizon execution.A separate termination head predicts whether the target view has been reached.Source: B. Anatomy-anchored Streaming Flow… · p. 6

1) Anchor-Relative Navigation Formulation:

  • A fixed episode anchor expresses each probe pose independently of the global measurement frame, creating a common representation for freehand demonstrations and robot roll-outs.The anchor-relative state also tracks scan progress from the anatomical starting point; task-specific landmarks include the xiphoid, T12, and wrist area.Source: 1) Anchor-Relative Navigation… · pp. 6–7

2) Multimodal Observation Encoding:

The policy encodes synchronized histories of ultrasound, wrist-camera, anchor-relative pose, and contact force. Force conditions trajectory generation, while the termination head uses a force-independent representation.

2) Multimodal Observation Encoding:

  • At each policy update, the model receives a synchronized history of K observations, including ultrasound, wrist-camera images, anchor-relative probe state, and normal contact force.Force measurements are synchronized with their corresponding ultrasound frames during training and robotic execution.Source: 2) Multimodal Observation Encoding · p. 7
  • Ultrasound, wrist-camera, and anchor-relative probe-state features are fused into cbase, while a separate force branch adds cforce to form trajectory condition ctraj.The resulting representation captures anatomy, local surface context, and scan progression relative to the initial anatomical reference.Source: 2) Multimodal Observation Encoding · p. 7
  • The termination head uses cbase without contact force, so stopping depends on ultrasound appearance, wrist-camera context, and anchor-relative probe state rather than instantaneous force variations.This separates force-sensitive motion generation from the stopping decision.Source: 2) Multimodal Observation Encoding · p. 7

3) Anchor-Relative Streaming Flow Learning:

The Streaming Flow Policy models probe motion in the anchor-relative pose space. It learns a stabilizing velocity field from demonstrations and repeatedly generates short trajectory chunks from updated observations.

3) Anchor-Relative Streaming Flow Learning:

  • The Streaming Flow Policy models probe-state evolution in anchor-relative pose space, with the trajectory condition formed from fused observations and a force residual.A termination head predicts whether the target view has been reached from the force-independent representation.Source: 3) Anchor-Relative Streaming Flow… · pp. 7–8, Figure 6 · p. 8
  • The policy parameterizes a time-dependent velocity field conditioned on ctraj over local progress τ within an H-step prediction horizon.Here τ runs from the beginning to the end of that horizon, not across the complete scanning episode.Source: 3) Anchor-Relative Streaming Flow… · p. 8
  • Training constructs a stabilizing target velocity by combining demonstrated trajectory motion with a correction that returns perturbed states toward the demonstration.The training procedure samples a trajectory index and a perturbed state around the corresponding demonstrated state.Source: 3) Anchor-Relative Streaming Flow… · p. 8
  • At inference, each rollout starts from the measured anchor-relative state, generates H future states with a fixed condition, and executes only the first Hexec states before querying again.The next rollout uses updated observations and the latest measured probe state, with local progress reset to zero.Source: 3) Anchor-Relative Streaming Flow… · p. 8

4) Termination Prediction:

A learned termination head predicts whether the target ultrasound view has been reached. Execution uses receding-horizon chunks, repeatedly incorporates new feedback, and constrains commanded pose changes.

4) Termination Prediction:

  • The termination head predicts target-view completion from detached, force-independent cbase, excluding contact-force features from the stopping decision.A positively weighted binary cross-entropy addresses the lower frequency of termination samples, and stop-gradient keeps its loss from updating the shared encoder.Source: 4) Termination Prediction · pp. 8–9

5) Receding-Horizon Policy Execution:

  • Only the first Hexec predicted states are executed before the policy is queried again with updated multimodal observations and the latest measured probe state.This receding-horizon procedure avoids executing the full predicted trajectory open-loop.Source: 5) Receding-Horizon Policy Execution · p. 9
  • Termination is confirmed only after the probability threshold is exceeded for consecutive policy updates; the robot then holds its measured probe pose.Before commands reach the controller, translational and rotational step limits and smoothing constrain target poses to prevent abrupt motions from isolated prediction errors.Source: 5) Receding-Horizon Policy Execution · p. 9

6) Post-Scan Best-View Selection:

After scanning ends, a separate module selects the preferred abdominal view from the recorded ultrasound sequence, rather than relying only on the frame at termination.

6) Post-Scan Best-View Selection:

  • After scanning terminates, a separately trained module retrospectively selects the preferred abdominal view from the recorded ultrasound sequence.This compares candidate views acquired during the scan instead of restricting the output to the frame observed at termination.Source: 6) Post-Scan Best-View Selection · p. 9
  • A frozen EchoCare encoder supplies pooled image features to organ-specific ranking heads trained on manually selected best-view frames.Training combines listwise cross-entropy and pairwise ranking losses, excluding temporally adjacent frames from comparisons.Source: 6) Post-Scan Best-View Selection · p. 9
  • At inference, a coarse-to-fine search scores every eighth frame before refining near the three highest-scoring candidates.The exported frame is the highest-ranked evaluated candidate; scores express relative preference within the video, not calibrated clinical image quality.Source: 6) Post-Scan Best-View Selection · p. 9

C. Anatomy-Guided Placement Module

The placement module uses external observations to locate a task-specific anatomical target, position the probe, and establish the anchor frame before learned navigation begins.

C. Anatomy-Guided Placement Module

1) Anatomical Target Localization:

  • For abdominal tasks, overlapping RGB views are combined into a whole-body image, and HSMR reconstructs the body and skeleton to identify task-specific anatomical points.Targets can be skeletal landmarks or points defined relative to multiple landmarks, adapting placement to body proportions.Source: 1) Anatomical Target Localization · p. 10, Figure 7 · p. 10
  • For radial-artery sweeping, MediaPipe Hands detects wrist landmarks from a forearm RGB-D image, avoiding whole-body skeletal reconstruction.The wrist landmark supplies the probe-placement target.Source: 1) Anatomical Target Localization · p. 10
  • The detected point is mapped through aligned depth into 3-D, transformed to the robot base frame, and associated with the nearest valid surface point.The anatomical model identifies the relevant location, while RGB-D supplies its metric position in the workspace.Source: 1) Anatomical Target Localization · p. 10

2) Placement Pose Generation:

  • The probe axis is aligned approximately perpendicular to the skin, while task-specific rotation sets its orientation; after contact, the probe-tip pose becomes the anchor frame.Local surface orientation is estimated from a neighborhood of surface points using principal component analysis.Source: 2) Placement Pose Generation · p. 10

A. Hardware Setup

  • The experiments assess the pipeline on a seven-degree-of-freedom robot setup with ultrasound, probe-footprint camera, whole-body camera, and force sensing.The study describes hardware and data acquisition before comparing policies, conducting ablations, testing automatic placement, and assessing transfer between a Panda and a KUKA robot.Source: A. Hardware Setup · p. 10, §IV · p. 10, Figure 8 · p. 11

B. Data Acquisition

The study collects multimodal demonstrations for three abdominal standard-plane tasks and one radial-artery sweep, then evaluates task success, image quality, stopping, efficiency, force, and motion smoothness.

B. Data Acquisition

  • The dataset contains 294 demonstrations across kidney, gallbladder, and liver standard-plane acquisition and radial-artery sweeping.Abdominal demonstrations begin with the probe footprint aligned craniocaudally; liver and gallbladder volunteers hold a deep breath during recording.Source: B. Data Acquisition · p. 11
  • Demonstrations record ultrasound, wrist-camera images, force, and probe poses; tracking failures are excluded and recordings are trimmed to valid imaging intervals.Each demonstration is annotated with the earliest frame at which the target plane is acquired or the prescribed sweep is complete.Source: B. Data Acquisition · p. 11

C. Evaluation Protocol

  • Task success is the proportion of attempted rollouts that complete the prescribed scan or acquire a qualifying abdominal view; artery success requires tracking over at least 95% of the scan extent.Task success is evaluated independently of autonomous stopping.Source: C. Evaluation Protocol · pp. 11–12
  • Six physicians rate abdominal target-view quality on a five-point scale, while radial-artery scan quality is measured by the vessel visibility ratio.The rated abdominal frame is selected by the same post-scan module for both methods, and physicians are blinded to the producing method.Source: C. Evaluation Protocol · p. 12
  • Time-to-target, peak normal contact force, and translational jerk characterize execution, while autonomous stopping success measures appropriate stopping after task completion without manual intervention.Continuous execution metrics and stopping performance are evaluated only on successful rollouts and reported as mean ± standard deviation.Source: C. Evaluation Protocol · p. 12

D. Navigation Policy Evaluation

On unseen participants, the anatomy-anchored policy outperformed fine-tuned π0 across four scanning tasks. It also achieved higher view-quality scores and more reliable stopping, with faster and smoother abdominal execution.

D. Navigation Policy Evaluation

  • Physician-rated ROI quality reached 3.47 for kidney, 3.97 for gallbladder, and 3.78 for liver, versus π0’s 2.78, 2.19, and 2.64, respectively.Radial-artery vessel visibility ratio was 99.6% for the proposed method versus 65.9% for π0.Source: D. Navigation Policy Evaluation · p. 12
  • 96.0% kidney, 94.0% gallbladder, 88.0% liver, and 94.0% radial-artery success were achieved on 16 unseen participants, exceeding fine-tuned π0 on all four tasks.Each method was evaluated over 50 rollouts per task; Table II compares the proposed policy with fine-tuned π0.Source: D. Navigation Policy Evaluation · p. 12, Table II · p. 13
  • 93.6–97.9% of successful abdominal rollouts and all successful radial-artery sweeps stopped autonomously, compared with π0’s 17.2–23.3% and 28.6%, respectively.The termination head uses ultrasound appearance, wrist-camera context, and anchor-relative progress.Source: D. Navigation Policy Evaluation · pp. 12–13
  • Mean time-to-target fell by 26–49% on abdominal tasks and 35% on radial-artery sweeping; abdominal peak contact force fell by 19–40% and translational jerk by 30–48%.For radial-artery sweeping, mean peak force was 4.4 vs. 2.9 N, with comparable jerk of 0.625 vs. 0.605 m/s3.Source: D. Navigation Policy Evaluation · p. 13
  • Representative rollouts pair ultrasound images with probe poses, showing navigation from task-specific anchors toward abdominal target views or along the radial artery.The examples include kidney progression from a T12 anchor through rib-shadowed views and task-specific navigation from the xiphoid for gallbladder and liver scans.Source: D. Navigation Policy Evaluation · pp. 13–14, Figure 9 · p. 13, Figure 10 · p. 14

E. Ablation Studies

Ablations show that wrist-camera input, force feedback, and anchor-relative poses each contribute to scanning performance. End-to-end trials achieve comparable task success with automatic placement, although kidney peak contact force increases.

E. Ablation Studies

  • Removing wrist-camera input reduces kidney success from 96.0% to 76.7% and radial artery success from 94.0% to 65.0%.Autonomous stopping also decreases from 97.9% to 78.3% for kidney and from 100% to 69.2% for radial artery sweeping.Source: E. Ablation Studies · pp. 14–15
  • Without force input, kidney peak force increases from 4.3 ± 2.0 N to 5.8 ± 5.5 N, while radial artery peak force rises from 4.4 ± 1.2 N to 5.4 ± 1.0 N.Fig. 11 shows a sharper early force rise and higher force over much of the scan without force input.Source: E. Ablation Studies · p. 15, Figure 11 · p. 15
  • With absolute poses, kidney success falls from 96.0% to 76.7% even in the nominal setup, and radial artery sweeping fails under rotation; anchor-relative performance remains largely unchanged.Under rotation, Anatomy-anchored SFP achieves 93.3% kidney success and 94.0% radial artery success.Source: E. Ablation Studies · pp. 15–16

F. Full System Validation

  • 90.0–100.0% success across four tasks with automatic placement, compared with 88.0–96.0% in scanning-only trials with manual placement.The reported rates were 100.0%, 93.3%, 90.0%, and 90.0% for kidney, gallbladder, liver, and radial artery tasks, respectively.Source: F. Full System Validation · p. 16
  • Kidney peak force rises from 4.3 ± 2.0 N with manual placement to 10.2 ± 6.2 N in the complete pipeline.The increase is associated with initial contact over the T12 spinous process.Source: F. Full System Validation · p. 16
  • Abdominal procedure times with automatic placement are 107.3 s for kidney, 116.2 s for gallbladder, and 116.3 s for liver scanning.These durations include initialization, including robot movement to two observation poses and landmark localization.Source: F. Full System Validation · p. 16

G. Cross-Embodiment Validation

Policies trained from freehand demonstrations transfer to a KUKA robot without retraining, with high success across four tasks. The conclusion also reports comparable end-to-end success with automatic placement and identifies validation on patients with pathology as future work.

G. Cross-Embodiment Validation

  • Transfer requires no adaptation beyond mechanical mounting and the end-effector-to-probe transformation; the probe is manually positioned at the initial scanning pose.The anchor-relative poses are composed with the anchor in the KUKA robot-base frame.Source: G. Cross-Embodiment Validation · p. 16
  • 83.3–100.0% task success on KUKA across four tasks without retraining, with average success of 90.8% versus 93.0% on Franka.KUKA execution is slower for kidney, gallbladder, and radial artery tasks than on Franka.Source: G. Cross-Embodiment Validation · pp. 16–17, Table V · p. 17

V. CONCLUSION

  • The framework combines anatomy-guided probe placement with learned navigation using anchor-relative motion, multimodal observations, and a termination head.Freehand demonstrations can be used directly for robotic execution without teleoperation or kinesthetic guidance.Source: §V · p. 17
  • Across unseen participants, task success is 88–96%; automatic placement gives comparable success to manual placement, and policies transfer to KUKA without retraining.The conclusion reports KUKA success of 83.3–100% across tasks.Source: §V · p. 17
  • Validation on patients with pathology remains future work; the reported validation is to be extended beyond healthy volunteers.The authors also plan to integrate the policy into an agentic framework.Source: §V · p. 17

Related papers

Loading 2610.32525v1…