PaperlayerView exact revision on arXiv

Source-linked AI summary

Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning

Noortje I. P. Schueler, Hans van Gorp, Ruud J. G. van Sloun

arXiv:2610.06008v1cs.LGcs.CV

TL;DR

Consistent ultrasound acquisition is important as demand grows and trained sonographers remain limited. The paper uses multi-step planning in a learned world model with retrieved scan transitions, and reports 83% target-view reachability in a real-time prospective CPU evaluation on unseen volunteers.

  • Problem

    Ultrasound acquisition depends on probe positioning, while growing examination demand and a limited supply of trained sonographers motivate guidance toward consistent standardized views.

    Source: §I · p. 1

  • Method

    The method plans several steps ahead in a learned latent representation, retrieving recorded transitions as a nonparametric world model and replanning after each new observation.

    Source: §I · pp. 1–2

  • Results

    83% target-view reachability was achieved in a real-time prospective evaluation on previously unseen volunteers using a CPU.

    Source: §I · p. 2

  • Takeaways & Limitations

    The prospective CPU evaluation on unseen volunteers provides a feasibility demonstration beyond retrospective simulation.

    Source: §I · p. 2

    The prospective evaluation was limited to three healthy volunteers and a single operator, and data came from one ultrasound system.

    Source: §V · p. 10

Abstract

from arXiv · show

Ultrasound is widely used, but acquisition quality is heavily dependent on the operator's knowledge and expertise. With demand for examinations outpacing the supply of trained sonographers, operator-guidance systems aim to close this gap by instructing a less trained user how to move the probe toward a target view. In this paper, we propose a retrieval-induced latent transition model for ultrasound acquisition dynamics, formulating ultrasound operator guidance as multi-step planning and retrieval in a world model. Using a V-JEPA 2.1 backbone, observations are first encoded into a latent space where anatomically related views lie close together. We then retrieve similar views from a reference database containing encoded latent states and corresponding probe positions and orientations. Rather than learning a parametric transition function, we directly use physically executed transitions from the database to establish our nonparametric, retrieval-induced transition model that supports receding-horizon planning. At deployment, guidance is generated from the live ultrasound image feed alone, without any probe tracking hardware. Applied to carotid ultrasound, the proposed planner reaches the target view in 86% of retrospective closed-loop episodes, versus 52% and 43% for representative baselines, outperforming both on every target view, including the challenging longitudinal internal and external carotid artery views. A prospective feasibility study on unseen volunteers, run in real time on a CPU using distillation, reaches 83% target-view reachability. Because planning is driven by proximity to any encodable goal latent, the same world model can navigate back to any previously acquired, patient-specific frame, supporting reproducible longitudinal imaging for e.g. perioperative or follow-up monitoring.

I. INTRODUCTION

The paper frames ultrasound guidance as multi-step planning in a learned world representation, using recorded transitions for retrieval-based planning rather than direct action prediction. It motivates this approach with the need for consistent target views and reports a prospective CPU feasibility result.

I. INTRODUCTION

  • Consistent acquisition of standardized views matters because image interpretation and quantitative analysis depend on acquired-data quality and consistency.The paper connects this need to growing examination demand and a limited number of trained sonographers.Source: §I · p. 1
  • Retrieving observed latent transitions lets the learned representation and scan trajectories serve as a world model without fitting a dynamics function.The retrieved transitions are used for planning rather than merely supplying context to a policy.Source: §I · p. 1
  • Ambiguous cross-sections may not identify location along a vessel, motivating actions that look several steps ahead rather than evaluating only one step.Prior guidance methods described here evaluate action consequences for at most a single step.Source: §I · p. 1
  • Carotid views vary across volunteers, while different views can look similar, making navigation challenging for people and AI guidance models.The task also requires full six-degree-of-freedom probe motion.Source: Figure 1 · p. 2, §I · p. 2
  • The method plans several steps ahead in latent space, selects a candidate trajectory, issues its first action, and replans after a new observation.Its transition model retrieves transitions recorded from real scans rather than learning a parametric transition function.Source: §I · p. 2
  • 83% target-view reachability was achieved in a real-time prospective evaluation on previously unseen volunteers.The evaluation used a CPU.Source: §I · p. 2

II. RELATED WORK

Prior ultrasound guidance ranges from direct action prediction to inference-time refinement, but the cited approaches do not compare multiple action sequences and their longer-horizon consequences at deployment. The paper distinguishes its approach by using retrieval as the transition model in an online planner.

II. RELATED WORK

  • Prior methods incorporate action consequences at different stages, from none in reactive guidance to training-time modeling and one-step refinement at inference.The paper identifies Cardiac Copilot as performing one-step inference-time action refinement.Source: §II · p. 2
  • One-shot systems issue predicted actions directly; examples predict rotation, while some autonomous carotid systems clone expert trajectories or select corrective motions.The cited methods do not evaluate what the world would look like after the predicted action.Source: §II · p. 2
  • Amortized approaches use consequences during training but not for explicit planning when guidance is issued.EchoWorld and Fan et al. use world-model objectives or imagined rollouts to shape representations or policies, then generate guidance through direct action regression or a trained policy.Source: §II · pp. 2–3
  • Cardiac Copilot queries a forward model at inference but refines one action at a time and restricts goals to predefined target embeddings.Its lookahead does not optimize an intermediate trajectory.Source: §II · p. 3
  • Retrieval-based planners beyond ultrasound reuse demonstrations, action sequences, or trajectory segments, often relying on a separate controller to execute selected subgoals or paths.The cited approaches include local regression, memory retrieval, topological search, and trajectory-segment recombination.Source: §II · p. 3
  • Learned world-model planners recursively predict action-conditioned latent futures, whereas this paper retrieves consequences from observed experience instead of generating hypothetical states with learned dynamics.The cited examples include visual-foresight approaches, DINO-WM, and V-JEPA 2-AC.Source: §II · pp. 3–4
  • The proposed planner recursively expands retrieved state-motion-state transitions into finite-horizon futures, scores them against the target and motion regularity, then issues only the first displacement before replanning.Retrieval therefore acts as a non-parametric action-conditioned transition model, rather than just a source of actions or subgoals.Source: §II · p. 4

A. General setup

The method frames ultrasound guidance as latent-space navigation using retrieved scan trajectories, while its dataset and representation are built to support that planning. The planner returns the first action of a selected trajectory and replans after each observation.

A. General setup

  • Guidance navigates from a live observation toward a target view by planning in latent anatomical states and retrieving candidate trajectories from prior scans.Only the selected trajectory’s first action is communicated, and planning repeats after each new observation.Source: A. General setup · p. 4

B. Dataset and implementation details

  • The training database is built from tracked freehand carotid scans, but deployment guidance requires no pose sensing.A robotic arm’s encoders synchronize six-degree-of-freedom probe poses with B-mode frames; the method uses relative pose displacements rather than absolute values.Source: B. Dataset and implementation details · p. 4
  • The dataset contains 247,827 frames from 48 volunteers, split by subject into 31 training, 4 validation, and 13 test trajectories.The six standard carotid views and a “no view” class form the label set; only 2.6% of training frames were labeled.Source: B. Dataset and implementation details · p. 4

C. Latent ultrasound world modeling

  • The latent space is intended to preserve anatomical similarity and probe-motion dynamics so retrieval can support meaningful transitions.The encoder retains the final tubelet from a four-frame clip, corresponding to the most recent two frames, and attentive pooling forms a compact state.Source: C. Latent ultrasound world modeling · p. 5
  • Training-only view-classification and dynamics heads shape the representation, but their predictions are not used for planning after training.Planning uses recorded database actions; a separate dedicated view-classification module supplies the view probabilities used for goal specification and trajectory scoring.Source: C. Latent ultrasound world modeling · p. 5
  • Each database state receives a view-membership probability from a ground-truth label when available or the dedicated classifier otherwise.This supports planning toward a desired standard view without requiring exemplar goal images.Source: C. Latent ultrasound world modeling · p. 5

D. Receding-horizon planning

The section introduces a nonparametric, retrieval-induced latent transition operator for representing realistic action-conditioned future states.

D. Receding-horizon planning

1) Retrieval-induced latent transition operator:

The transition operator uses nearby stored latent states as anchors and their recorded probe motions to form candidate transitions. Its set-valued structure offers multiple action-conditioned future states rather than a learned single transition.

1) Retrieval-induced latent transition operator:

  • The database contains acquisitions represented as sequences of latent states and corresponding probe poses.A stored acquisition and a candidate trajectory assembled during planning are distinct; candidate trajectories may combine acquisitions.Source: 1) Retrieval-induced latent transition… · p. 5
  • For a query latent state, the operator retrieves its 10 nearest database states by cosine similarity and samples physically observed forward or backward transitions from those anchors.At each anchor and selected frame stride, the planner chooses the successor with the highest immediate target-view probability and uses its associated relative probe motion.Source: 1) Retrieval-induced latent transition… · p. 6
  • The retrieval-induced operator is set-valued: each candidate transition pairs a retrieved similar state with a corresponding recorded probe motion and future state.Source: 1) Retrieval-induced latent transition… · p. 6

2) Recursive trajectory expansion:

The planner recursively expands retrieved transitions into candidate trajectories, then selects a trajectory and communicates only its first action. Retrieval and planning repeat from the next observation.

2) Recursive trajectory expansion:

  • At each planning depth, the planner retrieves a new neighbor set and expands candidate trajectories, using an adaptive stride set and a horizon of six.Higher target-view probability uses smaller strides, while lower probability uses larger strides.Source: 2) Recursive trajectory expansion · p. 6
  • Re-retrieval after each transition lets a candidate combine locally compatible recorded motions from different acquisitions into a trajectory absent from the database.Source: 2) Recursive trajectory expansion · p. 6
  • To limit search growth, trajectories are grouped by initial retrieved anchor and periodically pruned to five per lane using frontier-state target-view cost.The planner selects the retained trajectory minimizing J(τ), communicates only its first action, and repeats planning after a new observation; candidate ranking balances target-view likelihood, progress, and smooth motion.Source: 2) Recursive trajectory expansion · p. 6, 3) Trajectory scoring · p. 6

3) Trajectory scoring:

Candidate trajectories are scored by average target-view cost across the planning horizon, with additional penalties that favor smooth probe motion and reduce reversals during replanning.

3) Trajectory scoring:

  • A lower per-state target-view cost indicates greater confidence that a latent state corresponds to the desired view, and trajectory scores average this cost over the planning horizon.The frozen view-classification head assigns the probability used to compute the target-view cost.Source: 3) Trajectory scoring · p. 7
  • The target-view cost favors trajectories that remain close to the target-view manifold throughout the rollout.The remaining cost terms regularize predicted probe motion.Source: 3) Trajectory scoring · p. 7
  • Momentum and cross-step penalties encourage smooth motion within trajectories and reduce oscillatory reversals across replanning iterations.Both use 1−cosine similarity between dominant translation and rotation axes after rescaling; the planner selects the trajectory with the lowest total cost.Source: 3) Trajectory scoring · p. 7

A. Latent space analysis

The latent-space analysis compares V-JEPA 2.1 with alternative encoders across classification, retrieval, and motion-related properties, then examines view organization and volunteer generalization. The retrospective evaluation replays held-out acquisitions using nearest recorded poses, which introduces execution noise.

1) Linear probing:

  • The proposed V-JEPA 2.1 encoder outperforms EchoJEPA and ImageJEPA in view-classification accuracy and F1 scores.Table I reports the classification results.Source: 1) Linear probing · p. 7, Table I · p. 7

2) Local retrieval consistency:

  • Nearest-neighbor retrieval is more likely with V-JEPA 2.1 to return a frame sharing the query’s label than with either baseline.Purity is the fraction of the ten nearest test-set frames that share the query label.Source: 2) Local retrieval consistency · p. 7, Table I · p. 7
  • All encoders show comparable correlations for translational probe motion, while V-JEPA 2.1 shows the strongest correlation for rotation.The analysis compares latent-state cosine distances with recorded probe-motion magnitudes using Pearson and Spearman correlations.Source: 3) Transition consistency · p. 7, Table II · p. 7

3) Transition consistency:

  • The validation-set embedding clusters by view, with bridges between similar views, but not by volunteer, indicating generalization across volunteers.The two-dimensional t-SNE embedding uses cosine distance.Source: 4) Latent visualization · p. 7, Figure 3 · p. 8

B. Retrospective navigation evaluation

  • Retrospective navigation replays held-out, densely sampled acquisitions across six standard planes and a background no-view class, projecting each predicted pose to the closest recorded pose.This nearest-pose projection enables replay but adds noise to executed actions; the authors note that human operators also cannot execute proposed actions perfectly.Source: B. Retrospective navigation evaluation · pp. 7–8
  • The comparison contrasts direct action prediction, single-step planning, and retrieval-based receding-horizon planning with a six-step horizon.The horizon and cost weights were selected on the validation set.Source: 1) Navigation planning · p. 8

1) Navigation planning:

The proposed planner succeeds more often than both baselines across navigation rollouts, including harder target views. Ablations favor multi-step planning, the complete cost, and retrieval across the navigation database.

1) Navigation planning:

  • In episode evaluations, the proposed planner reached the target view in 86% of episodes, compared with 52% for UltraSeP-style guidance and 43% for Cardiac Copilot-style guidance.Its largest advantage was on the harder ECA and ICA targets, where it reached target views more often than the alternatives.Source: 1) Navigation planning · p. 8
  • Rotation backtracking rates are similar across methods, while UltraSeP-style guidance has higher translation backtracking than the other two methods.The backtracking rate counts consecutive nonzero action pairs with non-positive cosine similarity; lower values indicate less backtracking.Source: 1) Navigation planning · p. 8, Figure 5 · p. 8

C. Ablation studies

  • Multi-step planning substantially improves success over a greedy horizon-one planner, with gains plateauing from H = 5 onwards.H = 6 is selected as a compromise between performance and computational cost; among multi-step planners, backtracking is minimized around H = 6.Source: C. Ablation studies · p. 9, Table III · p. 9
  • Removing any trajectory-scoring term lowers success, and removing Jxstep also increases backtracking; the complete objective best balances success and smoothness.The ablation compares each term removal while keeping the remaining objective terms unchanged.Source: C. Ablation studies · p. 9, Table IV · p. 9
  • Planning across the navigation database yields higher success and lower backtracking than following temporal transitions within one best-retrieved acquisition.Database-wide retrieval can transition across acquisitions and volunteers when that produces a more promising trajectory.Source: C. Ablation studies · p. 9

D. Navigation to previously acquired patient-specific view

The planner can navigate to a previously acquired patient-specific frame, and prospective testing on unseen volunteers evaluates guidance for standard views. The prospective study reports 83% reachability for the proposed method.

D. Navigation to previously acquired patient-specific view

  • 79% success was achieved when navigating to patient-specific goal frames selected for 13 held-out subjects.Each target view had one selected goal frame and three attempts; starts were within 500 frames and at least 2 cm or 30° from the target, with performance assessed over 20 planning steps.Source: D. Navigation to previously acquired… · p. 9

E. Prospective feasibility study

  • The prospective feasibility study involved three unseen healthy volunteers and ran in real time on a standard CPU using a distilled V-JEPA 2.1 model.The examinations were conducted by the same operator who acquired the retrospective dataset; the distilled model used a planning horizon of three.Source: E. Prospective feasibility study · p. 9
  • 83% target-view reachability was achieved by the proposed method, versus 72% for the Cardiac Copilot-inspired method.Reachability required successful arrival in at least one of two attempts per target view.Source: E. Prospective feasibility study · p. 10

V. DISCUSSION

The discussion presents retrieval-based receding-horizon planning as effective for carotid navigation and flexible enough to target patient-specific frames. It also identifies limitations in retrospective simulation, annotation, scanner diversity, and prospective-study scope.

V. DISCUSSION

  • The framework uses a non-parametric retrieval-based transition model for receding-horizon planning and outperformed baselines in retrospective carotid navigation while remaining effective in prospective testing.The prospective feasibility study ran in real time on a standard CPU.Source: §V · p. 10
  • Navigation can target patient-specific frames, extending the framework beyond fixed clinical protocols.The discussion presents this as a flexibility of the framework.Source: §V · p. 10
  • Retrospective evaluation projects predicted poses onto the nearest recorded frame, coupling executed actions to available data; multi-rater annotation is also needed to assess agreement and view ambiguity.The authors note that real operators likewise cannot execute actions exactly.Source: §V · p. 10
  • Data came from one ultrasound system, and prospective testing covered three healthy volunteers and one operator; cross-scanner robustness and evaluation with multiple operators, including novices, remain future work.The authors also identify clear communication of continuous six-degree-of-freedom guidance as a challenge.Source: §V · p. 10

DATA STATEMENT

  • The study protocol was reviewed and approved by Eindhoven University of Technology's ethical review board, and all participants provided informed consent.Source: DATA STATEMENT · p. 10

Related papers

Loading 2610.06008v1…