PaperlayerView exact revision on arXiv

Source-linked AI summary

ROOM: A Physics-Based Continuum Robot Simulator for Photorealistic Medical Datasets Generation

Salvatore Esposito, Matías Mattamala, Daniel Rebain, Francis Xiatian Zhang, Kevin Dhaliwal, Mohsen Khadem, Subramanian Ramamoorthy

arXiv:2509.13177v2cs.RO

TL;DR

Bronchoscopy research lacks realistic training and test environments, and clinical data collection is constrained. ROOM uses patient CT scans to generate photorealistic, multimodal data at medical scales. Its evaluation covers pose and depth estimation, and the authors report that ROOM data can help fine-tune models to improve performance in real settings.

  • Problem

    Bronchoscopy data collection is constrained by patient-safety protocols, ethical review, and cost, while datasets must reflect anatomical variation and procedure-specific imaging and sensor requirements.

    Source: §I · p. 1

  • Method

    ROOM combines patient-specific anatomy, continuum-robot simulation, and photorealistic rendering to generate multimodal bronchoscopy data at medically relevant scales.

    Source: §I · p. 2, B. Continuum Robot Modelling · p. 3, C. Anatomical Reconstruction and Data… · p. 4

  • Results

    ROOM data contributes to addressing challenges in pose and depth estimation, and fine-tuning with the synthetic data improves performance in real settings.

    Source: §V · p. 6, §VI · p. 7

  • Takeaways & Limitations

    ROOM generates diverse training datasets that capture clinical-procedure complexity across patient anatomies and challenging scenarios.

    Source: §VI · p. 7

    The reconstruction pipeline depends on CT quality and may fail on severe occlusions or abnormal geometries; PyBullet may not fully reflect real-bronchia contact and deformable dynamics.

    Source: §V · p. 7

Abstract

from arXiv · show

Continuum robots are advancing bronchoscopy procedures by accessing complex lung airways and enabling targeted interventions. However, their development is limited by the lack of realistic training and test environments: Real data is difficult to collect due to ethical constraints and patient safety concerns, and developing autonomy algorithms requires realistic imaging and physical feedback. We present ROOM (Realistic Optical Observation in Medicine), a comprehensive simulation framework designed for generating photorealistic bronchoscopy training data. By leveraging patient CT scans, our pipeline renders multi-modal sensor data including RGB images with realistic noise and light specularities, metric depth maps, surface normals, optical flow and point clouds at medically relevant scales. We validate the data generated by ROOM in two canonical tasks for medical robotics: multi-view pose estimation and monocular depth estimation, demonstrating diverse challenges that state-of-the-art methods must overcome to transfer to these medical settings. Furthermore, we show that the data produced by ROOM can be used to fine-tune existing depth estimation models to overcome these challenges, also enabling other downstream applications such as navigation. We expect that ROOM will enable large-scale data generation across diverse patient anatomies and procedural scenarios that are challenging to capture in clinical settings. Code and data: https://github.com/iamsalvatore/room.

I. INTRODUCTION

Clinical data collection is constrained, while bronchoscopy requires anatomically faithful, procedure-specific imaging and calibrated sensor modalities. ROOM introduces an automated, photorealistic simulation pipeline for generating medically scaled data and demonstrates its use in medical-robotics tasks and further applications.

I. INTRODUCTION

  • The data-generation pipeline extracts a medial axis, samples branches more densely near bifurcations and high-curvature regions, synthesizes synchronized streams, then models sensor noise.Noise characteristics are matched to real bronchoscopy imagery through frequency-domain analysis.Source: Figure 2 · p. 2
  • ROOM transforms patient CT scans into synthetic training datasets while preserving geometric constraints and visual characteristics essential for medical navigation.The framework is designed for continuum robot bronchoscopy and generates data at medically relevant scales.Source: §I · p. 2
  • The pipeline produces RGB imagery with realistic noise, metric depth maps, surface normals, point clouds, and optical flow at bronchoscopy scales.Its rendering pipeline accounts for endoscopic lighting, tissue surface properties, and data-driven sensor models.Source: §I · p. 2
  • ROOM data is validated in multi-view pose estimation and monocular depth estimation, two medically relevant tasks.Source: §I · p. 2
  • The work also demonstrates monocular-depth fine-tuning and visual navigation applications using ROOM data.Source: §I · p. 2
  • ROOM is released as open source for community use.Source: §I · p. 2

II. RELATED WORK

Prior medical-robotics simulators often prioritize training or haptic feedback over vision-data generation, while existing endoscopy pipelines and bronchoscopy localization methods address different tasks. ROOM is positioned as a unified framework for visually accurate data collection in physically realistic settings.

Medical Robotics Simulators

  • Medical-robotics simulators often target surgical training and haptic feedback, with simplified visual rendering that is insufficient for training sim-to-real vision systems.Source: §II · p. 2
  • ORBIT-Surgical offers fast real-time rendering, but is not designed for large-scale dataset generation or multimodal sensor output; SimCol3D targets colonoscopy reconstruction and estimation tasks.Source: §II · p. 2

Continuum Robot Bronchoscopy Systems

  • Bronchoscopy’s appearance and geometric degeneracies motivate advanced rendering, while ROOM combines visually accurate data collection with physically realistic settings.Prior localization work includes PANS map-based 6-DOF tracking and an ex-vivo map-free visual-odometry dataset; the paper does not target these specific tasks.Source: §II · p. 2

III. METHOD

ROOM combines continuum-robot modelling, physics simulation, anatomical reconstruction and trajectory planning, and photorealistic rendering. Its robot model uses three clinically aligned controls and simulates contact and actuation non-idealities.

A. Overview

  • ROOM integrates robot modelling, physics simulation, anatomical reconstruction and trajectory planning, and photorealistic rendering with endoscopic artifacts.The visual comparison shows ROOM's Blender path-traced rendering reproducing tissue properties and lighting that naive PyBullet rendering lacks.Source: A. Overview · pp. 2–3, Figure 3 · p. 3

B. Continuum Robot Modelling

  • The bronchoscope uses three degrees of freedom: tendon displacement bends it, axial rotation selects the bending plane, and insertion controls depth.The model assumes constant strain along the flexible segment and uses an inextensible, unshearable rod reduction.Source: B. Continuum Robot Modelling · p. 3, Figure 4 · p. 3
  • A compliant PyBullet multibody simulation resolves external contact and friction around the nominal constant-curvature shape.The backbone is discretized into capsule collision bodies joined by revolute joints; the rod model itself does not impose contact as a boundary condition.Source: B. Continuum Robot Modelling · p. 3
  • Airway interaction uses Coulomb friction and compliant normal contact, while stochastic delays and magnitude-dependent scaling model actuation non-idealities.The contact model specifies static and dynamic friction coefficients of 0.3 and 0.25, respectively.Source: B. Continuum Robot Modelling · p. 3

C. Anatomical Reconstruction and Data Synthesis

ROOM reconstructs patient-specific airway geometry from CT, samples trajectories along the airway medial axis, and renders synchronized multimodal data with modeled sensor noise.

C. Anatomical Reconstruction and Data Synthesis

  • Patient CT volumes are resampled and normalized, segmented with TotalSegmentator's 3D U-Net, then converted to a signed distance field and watertight airway mesh.The mesh serves as rendering geometry and input to medial-axis extraction.Source: C. Anatomical Reconstruction and Data… · p. 3
  • Collision-free trajectories follow the airway medial axis extracted from the signed distance field, with denser sampling at bifurcations and high-curvature regions.An inverse-kinematics controller tracks waypoints to produce target 6-DoF poses at 10 Hz.Source: C. Anatomical Reconstruction and Data… · p. 4
  • Each target pose yields synchronized 600×600 RGB images, metric and relative depth, surface normals, optical flow, point clouds, and timestamped camera and robot metadata.Rendering uses physically based tissue materials and tip-mounted directional lighting; depth, normals, and flow come from Blender multi-pass rendering.Source: C. Anatomical Reconstruction and Data… · p. 4, Figure 5 · p. 4
  • Synthetic RGB noise is shaped to match the measured real-image power spectrum, then added to the rendered image with an amplitude parameter.The paper reports that matching noise statistics is crucial for assessing monocular depth estimation performance.Source: C. Anatomical Reconstruction and Data… · p. 4

IV. APPLICATIONS

ROOM data are demonstrated in pose estimation and depth estimation, with further demonstrations of depth-model fine-tuning and potential navigation applications.

IV. APPLICATIONS

  • ROOM-generated data are demonstrated in multi-view pose estimation and monocular depth estimation, two canonical medical-robotics tasks.Source: §IV · p. 4
  • The paper also demonstrates using synthesized data to fine-tune existing depth-estimation models.Source: §IV · p. 4
  • Navigation is identified as a potential application of the synthesized data.Source: §IV · p. 4

A. Task 1: Multi-View Pose Estimation

Airway branching and limited texture challenge multi-view pose estimation. On held-out sequences, VGGT outperforms the reported classical methods and DUSt3R on rotation and translation accuracy.

A. Task 1: Multi-View Pose Estimation

B. Task 2: Monocular Depth Estimation

Monocular depth is important in bronchoscopy, where instrument size constrains stereo or structured-light setups. ROOM evaluations show weak performance across existing models, with errors concentrated in challenging airway regions.

B. Task 2: Monocular Depth Estimation

  • Monocular depth matters because bronchoscope size constraints make stereo or structured-light setups challenging.Bronchoscopes range from 2.4–6.2 mm in outer diameter.Source: B. Task 2: Monocular Depth Estimation · p. 5
  • The evaluation compares general-purpose and endoscopy-specialized monocular depth models using L1, RMSE, absolute relative error, and δ accuracy thresholds.Relative-depth methods are scale-aligned per sequence using the ground-truth median depth.Source: B. Task 2: Monocular Depth Estimation · p. 5, Table II · p. 5
  • Across baselines, L1 was ≈9.0–11.0 mm and δ1 ≈27–31%; UniDepth led L1 (10.6 mm), while BREA-Depth led δ1 (30.8%).BREA-Depth's Abs Rel was 0.421; the reported natural-image δ1 benchmark is 80–90%.Source: B. Task 2: Monocular Depth Estimation · p. 5, Table II · p. 5
  • Errors cluster at specular highlights and geometric transitions; DAV2 errors are more diffuse, while Metric3DV2 and UniDepth retain structure but fail at boundaries.Limited texture also leaves smooth airway walls with broadly distributed errors.Source: B. Task 2: Monocular Depth Estimation · p. 5
  • ROOM's results indicate that bronchoscopy violates core assumptions of existing depth methods, motivating domain-specific training approaches to bridge the performance gap.Source: B. Task 2: Monocular Depth Estimation · p. 5

C. Task 3: Fine-tuning Monocular Depth Models

The authors fine-tune monocular depth models on ROOM-generated data and evaluate transfer to an external bronchoscope dataset. BREA-Depth improves across metrics, including on real bronchoscopy images.

C. Task 3: Fine-tuning Monocular Depth Models

D. Demonstration: Vision-Based Navigation

The paper demonstrates integrating ROOM outputs into a vision-based bronchoscope navigation pipeline. The example uses predicted depth for local mapping and collision checking, but does not quantitatively evaluate navigation.

D. Demonstration: Vision-Based Navigation

V. DISCUSSION

The authors conclude that ROOM data can help established pose- and depth-estimation methods address bronchoscopy challenges. They identify CT reconstruction and physical simulation as limitations and point to potential future uses.

V. DISCUSSION

  • ROOM's synthetic data can help established multi-view pose- and monocular-depth methods overcome challenges in the bronchoscopy domain.Source: §V · p. 6
  • The qualitative fine-tuning examples include real images, but those images have no depth ground truth.Source: Figure 7 · p. 7
  • Anatomical reconstruction depends on CT quality and may fail with severe occlusions or abnormal geometries; the simulator may not fully reflect real bronchial contact and deformation.The authors identify tissue deformation and respiratory motion as potential realism improvements.Source: §V · p. 7
  • The physical simulator may support future research on closed-loop navigation, traditional planner validation, and imitation- or reinforcement-learning policies.Source: §V · p. 7

VI. CONCLUSIONS

ROOM’s evaluation found that bronchoscopy poses challenges for existing methods, while ROOM-generated synthetic data can support fine-tuning that improves performance in real settings. The framework combines patient-specific anatomical reconstruction, continuum robot physics, and photorealistic rendering, and its modular architecture is expected to support component and scan substitutions.

VI. CONCLUSIONS

  • ROOM integrates patient-specific anatomical reconstruction, continuum robot physics, and photorealistic rendering at medically relevant scales to address data scarcity in bronchoscopy robotics.Source: §VI · p. 7
  • Evaluation in multi-view pose estimation and monocular depth estimation found bronchoscopy challenging for existing methods, while ROOM data can fine-tune them and improve performance in real settings.Source: §VI · p. 7
  • ROOM’s modular architecture is expected to let researchers test new CT scans or substitute rendering engines, physics simulators, and robot models.Source: §VI · p. 7
Loading 2509.13177v2…