PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Image-based Robotic Assistance and Guidance Environment (MIRAGE): A Novel Unified Framework for Surgical Navigation and Visualization

Ehsan Nasiri, Yi Zheng, Michael A. Kokko, Xiaoyao Fan, William R. Warner, Keith D. Paulsen, Ryan J. Halter

arXiv:2610.12780v1cs.RO

TL;DR

MIRAGE addresses whether heterogeneous imaging, tracking, reconstruction, and rendering can be integrated while retaining navigation accuracy and real-time responsiveness. It coordinates these streams in a unified framework, and phantom tests across four da Vinci platforms demonstrate technical feasibility and integrated navigation.

  • Problem

    Many image-guided robotic surgery systems are evaluated as isolated demonstrations; this paper asks whether heterogeneous streams can be synchronized while preserving accuracy and real-time responsiveness.

    Source: §1 · p. 2

  • Method

    MIRAGE coordinates heterogeneous imaging and tracking streams in a time-synchronized architecture integrating stereo reconstruction, AR overlay, HUD navigation, ultrasound, tracking, calibration, and registration.

    Source: §7 · p. 45

  • Results

    Phantom testing across da Vinci S, Si, Xi, and SP demonstrated technical feasibility, cross-platform deployment, real-time multimodal visualization, calibration accuracy, stereo reconstruction, and integrated navigation.

    Source: §7 · p. 46

  • Takeaways & Limitations

    Evaluation separates calibration, registration, overlay, reconstruction, and runtime endpoints to characterize the integrated workflow at component and system levels.

    Source: §7 · p. 46

    Phantom-based validation does not establish clinical effectiveness, patient benefit, or superiority; human factors, learning curves, clinical outcomes, and long-term monitoring remain unevaluated.

    Source: §6 · p. 45

Abstract

from arXiv · show

This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.

1 INTRODUCTION

MIRAGE integrates imaging, tracking, reconstruction, calibration, and visualization to overlay tracked preoperative CT anatomy on stereo endoscopic video in real time. The work evaluates its accuracy, responsiveness, and workflow as preclinical technical validation, not as evidence of clinical superiority.

1 INTRODUCTION

  • The contribution is coordinated, time-aware integration—not the individual use of established components—through explicit module states, timestamp association, transform propagation, synchronization rules, and evaluation metrics.A custom in-process C++/Python binding coordinates heterogeneous modules using buffer ownership, timestamped exchange, shared state, worker management, and copy-free frame access where possible.Source: §1 · pp. 3, 4, 6

2.2 Registration Pipeline

  • Registration and navigation chain component poses into a common tracker or CT frame, with optical and EM fiducials, saved transforms, and a background pipeline that updates models, instruments, and HUD metrics.The workflow describes established calibration and registration methods while focusing its integration analysis on coupled spatial, temporal, and configuration dependencies.Source: §1 · p. 5, §2.2.1 · p. 9, §2.2.2 · p. 9

1 INTRODUCTION

  • MIRAGE integrates stereo video, optical/electromagnetic tracking, augmented-reality overlays, stereo reconstruction, ultrasound, HUD navigation, and console visualization in one state-driven framework.Its workflow includes preoperative imaging, calibration, segmentation, and patient-specific modeling, followed by intraoperative tracking, registration, and selectable visualization modes.Source: §1 · pp. 3, 5, 6, §2.1 · p. 7

1.1 Research Question and Evaluation Hypotheses

  • The study asks whether heterogeneous modules can be integrated while preserving geometric accuracy and responsiveness, and frames hypotheses around transform accuracy, stream alignment, frame access, and reproducible evaluation.The proposed quantitative protocol covers calibration, registration, overlay and reconstruction errors, response time, synchronization error, and platform-specific repeatability; phantom validation spans da Vinci S, Si, Xi, and SP.Source: §1.1 · p. 6, §1 · p. 6

1.2 IDEAL-Aligned Evaluation Positioning

  • MIRAGE is positioned primarily at IDEAL stages 0–1, with early stage 2a elements; clinical outcomes, learning curves, human factors, failure modes, economics, and long-term surveillance remain future work.The evaluation is a preclinical technical and workflow validation, not evidence of clinical superiority or improved patient outcomes.Source: §1.2 · p. 7

2.3 Temporal Synchronization and Stream Association

  • Video, tracking, rendering, ultrasound, and GUI samples use a monotonic host-clock timebase; tracker poses are interpolated for video timestamps, and stale overlays are frozen or disabled by policy.The largest source-sample offset and interpolation bracket width describe temporal support, not physical synchronization error; acquisition, arrival, clock offset, and waiting contribute separately.Source: §2.3 · p. 10

2.4 Heads-Up Display (HUD)-TilePro

  • The HUD selects a target using distance and approach angle, highlights the best match, and displays tool-to-target guidance metrics for navigation.The scene and HUD refresh at rates up to 40Hz, supporting quantitative registration assessment and navigation guidance.Source: §2.4 · pp. 11–13

2.5 Augmented Reality (AR) Video Overlay

  • MIRAGE overlays segmented preoperative CT anatomy on the tracked endoscopic scene and supports surgeon-controlled visibility, mesh, and transparency modes.The framework was validated on kidney, throat, and surgical phantoms.Source: §2.5 · p. 13
  • Reprojection RMSE was approximately 6 pixels on da Vinci S cameras and 2 pixels on Xi cameras.The OpenCV calibration uses a custom checkerboard/retroreflector rig.Source: §2.5 · p. 14

2.5.1 System Overview

  • The system combines optical tracking, CT/tool registrations, calibrated stereo parameters, pose smoothing, OpenGL rendering, and SDI video I/O, with dedicated threads updating both eye views in real time.The GUI adjusts the overlay blend weight; pose smoothing, camera-frustum construction, transform composition, and rendering are implemented in MIRAGE.Source: §2.5.1 · pp. 14–15

2.6 Stereo Endoscope Calibration and Optimization Pipeline

  • Stereo calibration synchronizes checkerboard images with tracked endoscope and checkerboard poses, detects and refines corners, then fits eye-specific intrinsics, distortion, and tracked extrinsics by minimizing reprojection error.The projection and distortion model follows the established formulation in.Source: §2.6 · pp. 16–17

SE(3) Pose Optimization

The tracked-marker-to-eye extrinsic is parameterized as a six-dimensional twist and mapped to SE(3). Its residual block jointly updates intrinsic/distortion and pose parameters, with the logarithm map initializing the twist from the current transform.

SE(3) Pose Optimization

  • The fixed tracked-marker-to-eye extrinsic is parameterized by a six-dimensional twist and mapped to SE(3).The twist coordinates comprise rotation and translation for the selected eye.Source: SE(3) Pose Optimization · p. 17
  • The residual block jointly updates nine intrinsic/distortion parameters and six pose parameters.The logarithm map initializes the twist from the current transform.Source: SE(3) Pose Optimization · p. 17
  • The rotational and translational twist coordinates are three-dimensional, and the skew-symmetric operator is used in the formulation.The passage identifies these quantities as ω and vtwist for the selected eye.Source: SE(3) Pose Optimization · p. 17

Output and Evaluation

The section describes ultrasound streaming and stereo point-cloud reconstruction, including calibration evaluation, CT-space registration, and visualization. It reports model-dependent reconstruction trade-offs and calibration evaluation across working distances.

2.7 Ultrasound Streaming

  • Ultrasound input can come from DICOM files or a live OpenIGTLink stream, with ROI and landmark annotation, pseudo-volume surface extraction, and tracker-supported alignment to CT.Outputs include landmark CSV files, optional per-frame ROI masks, and 3D meshes; live streaming follows OpenIGTLink, DICOM metadata supplies physical coordinates and voxel spacing, and surface extraction uses marching cubes.Source: §2.7 · pp. 17–19

2.8 3D Stereo Reconstruction Point Cloud

  • Stereo reconstruction rectifies synchronized image pairs, estimates learned disparity, builds and cleans a colored point cloud, and displays it with preoperative CT meshes.Live optical-tracker poses update the phantom model pose so the meshes remain registered in CT space.Source: §2.8 · p. 19, Figure 9 · p. 24
  • RAFT-Stereo generally yields superior results with shorter runtimes, while PSMNet can be more accurate for some object-surface reconstructions.Both models are implemented in the framework and can be selected as needed.Source: §2.8 · p. 20

2.8.1 Calibration and Rectification; 2.8.3 Back-Projection to 3D

  • The stereo pipeline maps disparity from rectified images to 3D using the stereo reprojection matrix, under a rectified pinhole model with specified focal-length and principal-point assumptions.Point color is sampled from the rectified left image.Source: §2.8.1 · p. 20, §2.8.3 · p. 21

2.8.4 Validity Masking and Cleanup

  • Disparity cleanup applies bilateral filtering, a 99.8th-percentile cap, minimum-disparity and photometric thresholds, morphological opening, and statistical and radius outlier removal.These steps produce a valid point set for subsequent visualization.Source: §2.8.4 · pp. 21–22

2.8.6 Live Registration to CT Space; 2.8.7 VTK Scene and User Interaction

  • The reconstructed cloud is rendered with translucent CT meshes, while live tracked poses maintain CT-space alignment and user controls support capture and scene interaction.The capture UI is closed before the VTK window opens to ensure exclusive control of tracker streaming.Source: §2.8.6 · pp. 22–23, §2.8.7 · p. 23

2.8.8 Control Flow

  • The reconstruction control loop acquires a stabilized rectified stereo pair, computes and cleans the point cloud, streams tracker poses during visualization, and returns to capture when requested.The evaluation extends calibration across working distances on the Xi and SP robots.Source: §2.8.8 · p. 23, §2.8.9 · p. 23

2.8.9 Outputs and evaluations

  • Table 2 summarizes endoscope calibration results across working distances for the da Vinci Xi and SP, with the 100 mm Xi calibration error illustrated in Fig. 9b.The optimized intrinsics and extrinsics are exported in YAML, with per-frame and average reprojection RMSEs reported for each camera.Source: Output and Evaluation · p. 17, §2.8.9 · p. 23, Table 2 · p. 23

3 FRAMEWORK ARCHITECTURE:IMPLEMENTATION

MIRAGE divides real-time video acquisition, rendering, and hardware communication between a C++ backend and a Python frontend for registration, reconstruction, navigation, ultrasound, and GUI control. Its contribution is an architecture for synchronized, thread-safe cross-language data and state exchange, with documented boundaries for reproduction on other hardware.

3 FRAMEWORK ARCHITECTURE:IMPLEMENTATION

  • MIRAGE assigns video acquisition, rendering, calibration support, and selected hardware-facing modules to C++, while Python implements higher-level modalities and the orchestrating GUI.The design targets image guidance rather than robot actuation; the robot remains driven by its native clinical console.Source: §3 · p. 25

3.0.1 Cross-Language State and Buffer Binding

  • The binding architecture combines explicit buffer ownership, timestamped frame exchange, mutex-protected shared state, asynchronous workers, and a unified state machine rather than relying on pybind11 alone.The system exposes selected C++ runtime state to Python through the OverlayEngine interface.Source: §3 · p. 25, §3.0.1 · p. 26
  • Python can access current C++ frame buffers as zero-copy views, but their storage must remain valid and protected from conflicting writes during reads.Retaining a frame beyond its validity interval requires ownership protection or a copy; long-running C++ loops release the Python interpreter lock to keep GUI operations responsive.Source: §3 · p. 25, §3.0.1 · p. 27

3.0.2 Mathematics of Bindings

  • The binding formalism describes function mappings, buffer exchange, parameter updates, thread safety, asynchronous commands, and shared application-state evolution across languages.Asynchronous command instances are limited to one active worker; the mappings unify calls, buffers, and parameters under real-time constraints.Source: §3.0.2 · pp. 27, 30, 32

3.0.3 Reproducibility and Implementation Details

  • The architecture is designed for reproduction beyond the study hardware, but some proprietary hardware interfaces cannot be fully released.Its formulation, pseudocode, configuration schema, and device-independent evaluation protocol support implementations using alternative capture devices, trackers, or robotic platforms.Source: §3.0.3 · p. 32

4 MULTIMODAL FRAMEWORK: INTEGRATION

MIRAGE integrates HUD navigation, AR overlay, stereo reconstruction, ultrasound, rigid registration, and stereo calibration through a shared multimodal framework. Discrete modes, per-mode state, and a unified event loop organize the modules and their synchronized data flows.

4 MULTIMODAL FRAMEWORK: INTEGRATION

  • The framework combines HUD navigation, AR overlay, stereo reconstruction, ultrasound, rigid registration, and stereo calibration through defined interfaces and a unified pipeline.The listed modalities include both C++ backend and Python GUI components.Source: §4 · p. 33

4.1 Modes and State Switching

  • GUI modes are discrete configurations, and a selector maps the current mode and user action to the next mode.The available modes include HUD-TilePro, video overlay, stereo reconstruction, and ultrasound.Source: §4.1 · p. 33

4.2 Per-Mode Subsystem States

  • Each mode maintains its own subsystem state, bundling the signals it needs so updates and rendering remain isolated yet composable.Examples include tracker poses, phantom-model transforms, overlay frames, GUI controls, and console messages.Source: §4.2 · pp. 33–34

4.3 Unified Application State and Controller

  • The controller updates tracker-driven visualization using calibration state and supplies C++-generated frame buffers to the AR overlay.The same control pattern applies to other modalities.Source: §4.3 · p. 34

4.4 Cross-Modal Actions and Event Loop

  • A single dispatcher routes GUI and device inputs to the active mode and schedules updates and events across modalities.All modes share the unified event loop.Source: §4.4 · p. 35

4.5 Frame Synchronization and FPS Monitoring

  • Frame rate is averaged over an observation window using the frames received and the elapsed time.The supplied passage defines the measurement window but reports no FPS value.Source: §4.5 · p. 35

4.6 Summary of Unified Multimodal System

  • The system is modeled as a hybrid automaton: modes switch discretely while subsystems evolve under a shared event loop and state machine.Real-time data flows use thread-safe, zero-copy interfaces, and the Python GUI observes and controls the full state.Source: §4.6 · p. 35

5 QUANTITATIVE EXPERIMENTAL VALIDATION

MIRAGE was evaluated as a preclinical, system-level navigation platform across four da Vinci platforms and seven accuracy, synchronization, responsiveness, and repeatability endpoints. The experiments report platform-specific overlay and calibration errors, reconstruction comparisons, and runtime measures.

5.1 Comparative Positioning Against Existing Systems

  • Table 3 positions MIRAGE as providing AR overlay, ultrasound, and time synchronization, with 16.7 ms/stereo overlay and 200 ms/stereo reconstruction.The comparison spans system-level capabilities rather than isolated algorithms.Source: §5.1 · p. 38, Table 3 · p. 38

5.2 Fiducial Registration Accuracy

  • Registration RMSE is reported as mean ± SD across trials, and the original experiment logs stored RMSE feedback during registration.The supplied passage describes the reporting structure, not specific registration RMSE values.Source: §5.2 · p. 39, Table 4 · p. 39
  • Calibration errors were 0.1, 0.3, and 0.6 mm at 100, 300, and 500 mm; Xi optimization reduced 2D pixel error from 4.3 to 2.5 px and 3D spatial error from 3.0 to 0.6 mm.These are separate working-distance and optimization results.Source: §5.2 · p. 39, Table 5 · p. 40

5.3 3D Overlay Accuracy

  • For the right kidney, PSMNet's reconstruction error was 2.15 mm versus 2.25 mm for RAFT-Stereo; RAFT-Stereo had lower error on the other three listed structures.Typical inference times were approximately 0.20 s per stereo pair for RAFT-Stereo and 0.60 s for PSMNet.Source: §5.2 · p. 40, Table 6 · p. 40
  • For Si, left-eye contour checks averaged 0.47 ± 0.10 mm and right-eye checks 0.40 ± 0.20 mm, using the documented 0.20 mm/pixel scale.The three checks were 0.4, 0.4, and 0.6 mm for the left eye and 0.4, 0.2, and 0.6 mm for the right.Source: Figure 13 · p. 42

5.4 Temporal Synchronization and End-to-End Latency

  • A 60 FPS stream has a 16.7 ms paired-frame display latency, while reconstruction takes approximately 200 ms per stereo pair.Plots show pauses for registration or modality switching; changing from overlay to point-cloud reconstruction required restarting trackers.Source: §5.4 · p. 43, Table 8 · p. 44

5 QUANTITATIVE EXPERIMENTAL VALIDATION

  • At 100 mm, mean overlay errors were 6.0 px for S, 3.0 px for Si, and 2.5 px for Xi and SP.These values describe tested configurations; calibration, image format, and trial conditions remain relevant to comparisons.Source: Table 7 · p. 41, §5.4 · p. 44

6 LIMITATIONS AND TRANSLATIONAL EVALUATION REQUIREMENTS

This phantom-based study establishes preclinical technical and workflow validation, not clinical effectiveness or patient benefit. Further evaluation is needed before conclusions about clinical use or superiority over existing workflows.

6 LIMITATIONS AND TRANSLATIONAL EVALUATION REQUIREMENTS

  • The phantom-based evaluation does not establish clinical effectiveness, patient benefit, or superiority over existing workflows.It is preclinical technical and workflow validation.Source: §6 · p. 45
  • The evaluation lacks formal human-factors testing, surgeon workload analysis, learning curves, clinical outcomes, and long-term monitoring.Future translation should include structured usability studies, failure-mode analysis, patient-centered outcomes, and real-world surveillance.Source: §6 · p. 45
  • Platform-specific validation is limited by experiment counts per da Vinci system, especially SP.The paper also identifies the need for latency testing under clinical workflow conditions and stale-transform and overlay-failure safeguards.Source: §6 · p. 45

7 CONCLUSION

MIRAGE integrates heterogeneous imaging and tracking streams into a time-synchronized robotic surgery navigation workflow. Phantom experiments across four da Vinci platforms support its interpretation as a preclinical technical and workflow-validation platform.

7 CONCLUSION

  • MIRAGE coordinates stereo reconstruction, AR overlay, HUD-TilePro navigation, ultrasound, tracking, calibration, and registration in a time-synchronized framework.Its contribution is the integrated, quantitatively evaluated architecture rather than an isolated calibration, registration, or reconstruction algorithm.Source: §7 · p. 45
  • At startup and after calibration, the framework detects and configures tracking hardware, loads platform and imaging-source settings, and launches its processing and visualization engine.The launch enables data exchange and control between low-level modules and the user interface.Source: §7 · p. 45
  • The interface lets users control live stereo overlays, select anatomical regions, view tracked distance and angle readouts, display imagery through TilePro, and capture video or frames.Modality-specific panels provide visualization and control.Source: §7 · p. 45
  • Phantom experiments on da Vinci S, Si, Xi, and SP support technical feasibility, cross-platform deployment, and integrated navigation; MIRAGE remains a preclinical validation platform.The evaluation separates calibration, registration, overlay, reconstruction, and runtime endpoints.Source: §7 · p. 46

8 FUTURE WORKS

Planned work extends MIRAGE's modalities and calibration and reconstruction capabilities, then evaluates usability and workflow through staged studies. The submission does not provide a public repository for raw acquisition logs or the complete implementation.

8 FUTURE WORKS

  • Planned technical extensions include MEWER depth mapping, related video streams, video-based calibration, continuous point-cloud reconstruction, and automatic overlay-quality assessment.The next calibration iteration is intended to use continuous video for calibration monitoring and real-time point-cloud updates.Source: §8 · p. 46
  • Future evaluation is planned to progress from simulated-procedure studies of usability and workload to learning-curve, safety, patient-centered, economic, and long-term assessments.Proposed technical studies include stale-transform detection, overlay-failure alerts, dropped-frame behavior, registration drift, and controlled timing and frame-transfer comparisons.Source: §8 · p. 46

DATA AVAILABILITY STATEMENT

  • The submission does not provide a public repository for raw acquisition logs or the complete implementation, which includes proprietary hardware interfaces.Source: DATA AVAILABILITY STATEMENT · p. 47

A.1 Optical Tracker Registration

MIRAGE registers CT fiducials to optical and electromagnetic tracker frames using corresponding point sets and rigid-transform fitting. The workflow saves registration quality feedback and composes tracker transforms with explicit source and destination frames.

A.1 Optical Tracker Registration

  • CT fiducials and corresponding optical stylus-tip positions are displayed for editing before rigid registration computes the CT-to-tracker transform and RMSE.The GUI saves the resulting transformation and RMSE.Source: §A.1 · pp. 48–49

A.2 Optical and Electromagnetic Registration Workflow

  • The optical and electromagnetic workflows fit rotations by SVD, correct improper rotations, and compute translations from the acquired fiducials.The EM fit yields TFG←CT; its inverse is used for tracker-to-tracker composition, with fitted and inverse transforms retaining distinct frames.Source: §A.2 · pp. 49–50

A.3 Real-Time Navigation Filtering

The framework smooths tracker poses while preserving valid rotations, and associates tracker data with video timestamps for overlay rendering. It also specifies behavior when valid pose data are unavailable and describes the stereo overlay pipeline.

A.3 Real-Time Navigation Filtering

  • After tracker poses are composed with saved CT registrations, element-wise smoothing is followed by projection of the rotational block onto SO(3), and filtered poses update CT-frame models, instruments, and targets.If no valid pose exists within Δtmax, the overlay is marked stale and frozen at the last valid transform or disabled under the safety policy.Source: §A.3 · p. 51, §A.4 · p. 52

A.4 Temporal Synchronization and Stream Association

  • Video and tracker samples are timestamped on a monotonic host clock, with tracker translation linearly interpolated and rotation interpolated by SLERP for each video timestamp.An interpolated pose assigned the video timestamp does not independently establish physical synchronization accuracy.Source: §A.4 · pp. 51–52

B.1 Overlay Pipeline

  • The overlay pipeline loads tracker poses, CT models, and camera calibration, then smooths poses, renders each eye independently, and blends virtual frames with video.GUI controls include mesh visibility, alignment offsets, opacity, and display settings.Source: §B.1 · p. 53

B.2 Optical Tracking and Pose Smoothing

  • Tracked tool poses are transformed through the CT registration and tracker-to-eye transforms, then exponentially smoothed with rotation projected back to a rigid rotation.The previous-estimate weight αAR is given as an example value of 0.9.Source: §B.2 · p. 53

Appendix A.3.

The appendix details camera projection and stereo rendering, concurrent execution, and checkerboard-based calibration. It specifies an explicit distortion model and an optimization implementation that exports calibrated parameters and reprojection errors.

B.3 Camera Projection and Rendering

  • Camera projection uses the inverse left-eye pose in CT space and converts optical coordinates to OpenGL axes before defining the frustum for anatomical-model rendering.The corresponding left-eye intrinsics and viewport dimensions provide the projection parameters.Source: §B.3 · p. 54

B.4 Real-Time Execution

  • Three concurrent threads handle stereo rendering, tracker queries and pose updates, and video-frame transfer, supporting input-video, virtual-render, and fused-overlay modes for both eyes.The video-I/O thread uses a Bluefish444 KRONOS K8 card.Source: §B.4 · p. 55

B.5 Checkerboard Acquisition and Projection

  • Calibration records stereo images alongside tracker poses for the endoscope and checkerboard, then detects and refines checkerboard corners for projection.The observations are generated after preprocessing with CLAHE, corner detection, and subpixel refinement.Source: §B.5 · p. 55

B.6 Explicit Reprojection Model

  • The reprojection model predicts pixel coordinates from camera-frame 3D points using focal lengths, principal point, and eye-specific distortion coefficients.The scalar formulas use the same eye-specific intrinsics and distortion parameters as the calibration vector.Source: §B.6 · p. 56

B.7 Optimization Implementation

  • Calibration optimization forms pixel-wise residuals between reprojected checkerboard points and observed corners, jointly parameterizing intrinsics and distortion with the fixed marker-to-eye pose.Optimized intrinsics and extrinsics are exported in YAML with per-frame and average reprojection RMSE.Source: §B.7 · pp. 56–57

C ULTRASOUND STREAMING DETAILS

The ultrasound module supports offline DICOM stacks and live OpenIGTLink frames, with tools for image annotation and mesh generation. Its coordinate and reconstruction procedures distinguish metadata-based physical mapping from pseudo-volume assembly without tracked probe motion.

C.1 Acquisition

  • Ultrasound acquisition supports multi-frame or per-frame DICOM files and live OpenIGTLink IMAGE messages, timestamping received frames and converting intensities to 8-bit for display when needed.The GUI maintains a recent cine buffer.Source: §C.1 · p. 57

C.2 Threading and Latency Control

  • Live frames enter a bounded queue that discards the oldest pending item when full, while the GUI polls about every 1 ms and renders the most recent item from each poll cycle.Source: §C.2 · p. 57

C.3 DICOM and Physical Coordinates

  • DICOM orientation and origin metadata map voxel indices into LPS coordinates using slice spacing; nonuniform or nonparallel stacks require individual slice positions and orientations.Visualization coordinates are converted to centimeters, with translations and model coordinates converted to the same unit before rigid-transform composition.Source: §C.3 · pp. 57–58

C.4 ROI Definition and Binary Mask

  • An ROI polygon defines a binary image mask, which can be smoothed by morphological opening and hole filling.The structuring-element radius controls the morphological operation.Source: §C.4 · pp. 58–59

C.5 Interactive Landmarks

  • Each ultrasound landmark stores its frame index, image coordinates, and user-selected anatomical label, and the list can be exported as CSV.Source: §C.5 · p. 59

C.6 Pseudo-Volume Assembly

  • Marching-cubes reconstruction assembles a stacked or cine pseudo-volume, but true 3D compounding requires external probe pose information.For cine-buffer reconstruction, the number of pseudo-volume slices equals the cine-buffer frame count.Source: §C.6 · p. 59

C.7 Control Flow and Outputs

  • The control flow supports loading stored stacks or live cine data, displaying frames, exporting ROIs and landmarks, and rendering 3D meshes.Outputs include landmark CSV files, optional per-frame ROI masks, and 3D meshes; physical dimensions come from DICOM metadata.Source: §C.7 · pp. 59–60, Table 9 · p. 60
Loading 2610.12780v1…