Ultrasound-Guided Robotic Navigation with Deep Reinforcement Learning

• Ultrasound is useful for real-time, radiation-free imaging but is difficult to navigate autonomously because of artifacts, low signal-to-noise ratio, subject dependence, and inter-observer variability.Source: §I · p. 1
• Existing robotic spinal navigation can combine force data and ultrasound images but still requires technician setup.Source: §I · p. 1
• Automatic navigation to specific positions on the human body without intervention remains unresolved.Source: §I · p. 1
• The authors pursue fully automated spinal navigation using ultrasound images for decision making through a combination of reinforcement learning and supervised learning.Source: §I · p. 1
• The study contributes a robot-tracked lower-back ultrasound dataset from volunteers and trains an RL agent in simulated environments to find correct sacrum views using ultrasound frames.Source: §I · p. 1

A. Deep Reinforcement Learning

• The acquisition setup attaches an ultrasound probe to the robot end-effector and stores ultrasound frames with robot tracking data.Source: Figure 1 · p. 2

B. Reinforcement Learning for Robotic Manipulation

• Prior visual robotic manipulation work found simulated RL results promising but reported failure when transferring the approach to real-world applications.Source: B. Reinforcement Learning for Robotic… · p. 2

C. Reinforcement Learning in Medicine

• Medical RL has been used for diagnostic decision support, where supervised learning assesses confidence and RL recommends further measurements when confidence is insufficient.Source: C. Reinforcement Learning in Medicine · p. 2
• Earlier medical visual RL guided users toward clinically relevant cardiac ultrasound views, while this work closes the navigation loop with a robotic actuator.Source: C. Reinforcement Learning in Medicine · p. 2
• The method combines DQN memory with a binary stopping classifier to improve navigation beyond the earlier user-guidance setup.Source: C. Reinforcement Learning in Medicine · p. 2

A. Reinforcement Learning

• Ultrasound-only decision making makes the navigation problem a partially observable Markov decision process because the agent must estimate its state from each frame.Source: A. Reinforcement Learning · p. 2

B. Deep Q-Learning

• The RL agent learns from replayed transitions containing states, actions, rewards, and next states, updating Q-values through discounted temporal-difference learning.The optimal action is selected by maximizing Q(s, a), with α denoting learning rate and γ denoting discount factor.Source: B. Deep Q-Learning · p. 3
• DQN training uses a target network, dueling value and advantage streams, and prioritized replay to improve stability and focus learning on informative transitions.The target network reduces changing-target instability, while prioritized replay samples transitions according to TD-error.Source: B. Deep Q-Learning · p. 3
• V-DQN estimates Q-values from the current ultrasound frame using feature extraction followed by value and action-advantage estimators.The model defines the current frame as I_t and uses separate value and advantage functions.Source: B. Deep Q-Learning · p. 3, Figure 2 · p. 3
• M-DQN adds previous frames and previous actions as two input streams so Q-value estimation incorporates recent interaction history.Previous actions are represented as concatenated one-hot vectors and combined with extracted frame features.Source: B. Deep Q-Learning · pp. 3–4
• MS-DQN adds a binary classifier for stopping, removing the stopping decision from the reward function to address sparse goal-bin examples.Without the classifier, stopping remains part of the Q-value layer as a fifth action value.Source: Figure 2 · p. 3, B. Deep Q-Learning · p. 4

C. Problem setting

• The agent must find the sacrum using only ultrasound information while navigating the spinal region.The search occurs in a defined parallel plane with two degrees of freedom.Source: C. Problem setting · p. 4
• The V-DQN and M-DQN action space includes up, down, left, right, and stop, whereas MS-DQN triggers stopping through its binary classifier.Thus, stopping is represented as an action in the former models but as a separate decision in MS-DQN.Source: 1) Action space · p. 4

2) State:

• The state is defined by the probe’s position relative to the sacrum in the parallel plane, satisfying the Markovian formulation used for the environment.Position alone fully defines the state in this problem setting.Source: 2) State · p. 4

3) Observation:

• Ultrasound frames serve as observations in a POMDP, with anatomical and acquisition differences creating observation variability across subjects.An agent that estimates state from these observations can select actions for navigation toward the goal.Source: 3) Observation · p. 4

4) Reward function:

• The reward function labels sacrum-containing bins as correct and penalizes incorrect stopping and repeated back-and-forth movement.Incorrect stopping terminates exploration in the wrong position, while oscillation accumulates net negative reward.Source: 4) Reward function · p. 4, Table I · p. 4

5) Simulated robot navigation implementation:

• Simulated tests initialize environments at predetermined states, provide the corresponding ultrasound frames, and execute the agent’s selected actions.The simulation follows the procedure described in Algorithm 1.Source: 5) Simulated robot navigation… · p. 4
• The navigation procedure terminates when the sacrum is reached.Algorithm 1 explicitly marks the sacrum-reached condition as the stopping point.Source: 5) Simulated robot navigation… · p. 4
• The simulated environments use projected ultrasound-image grids whose frames vary across subjects and anatomical regions.The displayed samples cover the left posterior pelvis, L3 vertebra, sacrum, and right lumbar region.Source: Figure 3 · p. 5

A. Project setup

• For data acquisition, a 7-axis KUKA LBR iiwa 7 R800 manipulator certified for human interaction is used, with robot control running on the Robotic Operating System (ROS)1.Source: A. Project setup · p. 5
• The ultrasound probe is mounted on the robot end-effector, with frames captured at 800x600 pixels and 30 fps.An Epiphan DVI2USB 3.0 frame-grabber receives the ultrasound frames.Source: A. Project setup · p. 5
• Imaging uses a L8-3 linear transducer at 70 mm depth and 90% gain, while compliant control limits z-axis force to 2 N.Source: A. Project setup · p. 5

B. Data-set

• Scans from 34 volunteers are mapped into 11x15 grids, with five frames per bin to simulate x-y navigation environments.Each scan contains eleven spine-parallel sweeps divided into 15 segments.Source: B. Data-set · p. 5
• Training uses 25 subjects with varied acquisition quality, while validation and testing use nine high-quality subjects split into four and five subjects.Source: B. Data-set · p. 5

1) Framework setup:

• The framework extends RL-zoo and stable-baselines using TensorFlow.The implementation is publicly available on GitHub.Source: 1) Framework setup · p. 5

2) Model Training:

• Each training episode starts from a random environment and permits 50 steps, with epsilon-greedy exploration decaying to 0.02 after one-third of training.Episodes terminate when the agent stops or reaches the step limit; epsilon-greedy exploration addresses the exploration-exploitation dilemma.Source: 2) Model Training · pp. 5–6
• The stopping classifier separates sacrum views from other frames, balances classes through oversampling and augmentation, and exceeds 99% test accuracy.Source: 2) Model Training · p. 6
• A DenseNet-121 classifier [25] predicts the navigation action for the supervised-learning baseline.Source: 2) Model Training · p. 6

3) Metrics:

• Policy correctness measures the proportion of actions directed correctly toward the goal across runs and test environments.Testing initializes agents in all 165 states of unseen environments and allows 20 actions per run.Source: 3) Metrics · p. 6
• Reachability counts runs that reach a goal bin and make a stopping decision, so reaching the sacrum without stopping is unsuccessful.The example run reaches the sacrum with g(e, r) = 1 after five correct actions.Source: 3) Metrics · p. 6

V. RESULTS AND DISCUSSION

• The proposed MS-DQN substantially outperforms other architectures by 20–30% in policy correctness and 40–60% in reachability.The authors attribute the improvement primarily to the binary stopping classifier, addressing the difficulty and sparsity of successful stopping transitions.Source: §V · p. 7
• The M-DQN improves reachability over the V-DQN by 20%, whereas the V-DQN exceeds the M-DQN in policy correctness by 6%.The authors associate memory with escaping loops but also with occasional sub-optimal paths.Source: §V · p. 7
• MS-DQN produces state-value estimates with a range similar to the training-environment ground truth, unlike the other DQN setups.Accurate state-value estimation is difficult in unseen environments, although advantage values can still support correct actions.Source: §V · p. 7, Figure 6 · p. 7
• Pure RL struggles with reward sparsity and performance in unseen environments, motivating further combinations of RL with supervised learning.The prioritized replay memory alone does not resolve the shortcoming associated with sparse successful stopping transitions.Source: §V · p. 7

• The proposed reinforcement learning-based ultrasound-guided robotic navigation method outperformed DQN and classification baselines despite substantial anatomical variability.The task involved spinal navigation to locate the sacrum.Source: §VI · p. 7
• Adding a binary classifier to decide when to stop substantially improved the method.Source: §VI · p. 7
• Online implementation in a medical setting would require ethical approval.Source: §VI · p. 8