PaperlayerView exact revision on arXiv

Source-linked AI summary

Markerless Multi-Modal Autonomous Robotic Inspection of Large Space Structures

Juan De Dios Alfaro, Arturo Ríos, David Rodríguez-Martínez, Carlos Pérez-del-Pulgar

arXiv:2609.29644v1cs.RO

TL;DR

Future orbital structures make inspection based on known geometry, cooperative markers, or predefined interfaces less suited to large, heterogeneous, or partially unknown targets. This paper integrates multimodal sensing, autonomous planning, and 3D reconstruction into a markerless robotic inspection pipeline. Simulation and preliminary laboratory validation show spatially coherent inspection data and reconstructions for visual and geometric assessment.

  • Problem

    Future structures may be large, heterogeneous, or partially unknown, while current servicing often assumes known geometry, markers, interfaces, or planned sequences.

    Source: §I · p. 1

  • Method

    The paper integrates a Kinova Gen 2 manipulator, multimodal sensing, LiDAR-based volume estimation, viewpoint and motion planning, synchronized ROS 2 logging, and 3D reconstruction.

    Source: §I · p. 1, §VIII · p. 7

  • Results

    Gazebo simulation and laboratory experiments validate complementary parts of the architecture, while Nerfacto and Thermal-Nerfacto produce spatially coherent geometric and thermal inspection outputs.

    Source: §VIII · p. 7

  • Takeaways & Limitations

    The resulting inspection data support visual and geometric assessment and represent a step toward inspection of larger, more complex non-cooperative structures.

    Source: §VIII · p. 7

    The fixed-base physical experiment limits field of view and attainable coverage, and the validation does not reproduce orbital dynamics, illumination, or operational constraints.

    Source: D. Discussion · p. 6

Abstract

from arXiv · show

Future orbital infrastructures, such as deployable antennas, solar farms, and large orbital platforms will require autonomous inspection systems able to operate with limited prior knowledge and without cooperative markers. Current on-orbit servicing approaches often rely on predefined trajectories, standard interfaces, fiducial markers or accurate target models, which limits scalability for large, heterogeneous or partially unknown structures. This paper presents a markerless autonomous robotic inspection pipeline in which 3D reconstruction is used as an inspection-support representation. The system integrates a Kinova Gen2 manipulator with an end-effector-mounted multimodal sensor head composed of an RGB-D camera, a thermal camera and a 2D LiDAR. The pipeline estimates an approximate inspection volume, generates viewpoints, plans collision-free motions with MoveIt, and synchronously records RGB-D images, thermal data, and robot poses in ROS2. Candidate reconstruction methods were evaluated to select a practical method for this pipeline, with Nerfacto used for geometric reconstruction and Thermal-Nerfacto used to demonstrate thermal-aware rendering for inspection. Validation in a Gazebo-based simulator and preliminary laboratory tests reveal that the proposed system can autonomously acquire spatially coherent inspection data and produce reconstructions suitable for visual and geometric assessment, representing a step towards inspection of large non-cooperative space structures.

I. INTRODUCTION

Future orbital structures make inspection based on known geometry, fixed interfaces, and preplanned operations less scalable. This paper proposes a markerless pipeline that estimates target volume, plans viewpoints, and uses 3D reconstruction as an inspection-support representation.

I. INTRODUCTION

  • Future large, modular structures require frequent inspection, maintenance, repair, and reconfiguration, while relying only on human operators or preplanned trajectories becomes less scalable as complexity grows.The stated examples include space stations, large antennas, deployable solar arrays, orbital construction platforms, and orbital data centres.Source: §I · p. 1
  • Current servicing often assumes known target geometry, predefined interfaces, fiducials, or planned sequences, limiting flexibility for large, heterogeneous, partially unknown structures.The paper describes these assumptions as suitable for specific missions but less suited to future environments.Source: §I · p. 1
  • Markerless inspection requires estimating an approximate target volume, selecting viewpoints, acquiring multi-perspective data, and generating a spatially coherent representation for visual and geometric inspection.The introduction also identifies low texture, repetitive patterns, reflections, and illumination changes as perception and reconstruction challenges.Source: §I · p. 1
  • The paper integrates reconstruction into a markerless inspection pipeline combining a Kinova Gen 2 arm, RGB-D, thermal and LiDAR sensing, viewpoint generation, MoveIt planning, and synchronized ROS 2 logging.The work also evaluates reconstruction methods and uses Gazebo simulation and laboratory experiments to test the system workflow.Source: §I · p. 1

A. Robotic Inspection and On-Orbit Servicing

  • Earlier servicing approaches commonly use cooperative targets, predefined interfaces, known geometries, or planned procedures; this work instead uses perceived geometry to guide markerless inspection and autonomous data acquisition.The related-work discussion frames adaptation through onboard perception and autonomous planning as relevant to future large structures.Source: A. Robotic Inspection and On-Orbit… · pp. 1–2

B. 3D Reconstruction for Robotic Inspection

3D reconstruction can support geometric analysis, coverage checking, and later diagnosis or repair planning, but candidate methods differ in their scene assumptions and pipeline needs. The paper treats reconstruction methods as integration candidates rather than isolated algorithms.

B. 3D Reconstruction for Robotic Inspection

  • Reconstructed models can support geometry analysis, coverage verification, novel views of inaccessible areas, and later diagnosis or repair planning.The paper positions reconstruction as a spatial basis for inspection tasks.Source: B. 3D Reconstruction for Robotic… · p. 2
  • Structure-from-Motion and Multi-View Stereo can recover sparse and dense models, while COLMAP depends on sufficient texture and image overlap for robust poses and dense reconstruction.The discussion notes that space-like scenes may instead have low texture, reflections, harsh illumination changes, and repeated patterns.Source: B. 3D Reconstruction for Robotic… · p. 2
  • NeRF methods represent scenes continuously from posed images; Nerfacto offers a practical modular implementation, while Gaussian Splatting, NeuS, CasMVSNet, and MASt3R target different rendering or geometric tasks,,,,,.The cited method descriptions distinguish real-time rendering, smooth surface recovery, depth estimation, and robust correspondences or 3D grounding.Source: B. 3D Reconstruction for Robotic… · p. 2
  • Reconstruction candidates are assessed for quality, robustness, integration ease, computational feasibility, and compatibility with the generated RGB-T-D dataset, not quality alone.The paper frames the methods as candidates for an integrated ROS 2 robotic workflow.Source: B. 3D Reconstruction for Robotic… · p. 2

C. Multimodal Perception in Space Robotics

The paper combines complementary sensing for inspection and evaluates synchronized thermal data alongside RGB-based geometric reconstruction. Its ROS 2 workflow carries sensing through viewpoint planning, motion execution, synchronized acquisition, and reconstruction.

C. Multimodal Perception in Space Robotics

  • RGB appearance, depth measurements, LiDAR-based spatial estimation, and thermal information offer complementary inputs for robotic inspection.Thermal imagery is presented as a potential diagnostic cue for anomalies such as overheating, insulation defects, or material degradation.Source: C. Multimodal Perception in Space… · pp. 2–3
  • The work evaluates synchronized thermal data through a thermal-aware neural reconstruction experiment, alongside geometric reconstruction mainly supported by RGB images and poses.The platform is presented as suitable for future thermal mapping, anomaly detection, and combined geometric-thermal inspection.Source: C. Multimodal Perception in Space… · p. 3

A. Overview

  • The ROS 2 workflow combines perception, autonomous viewpoint generation, motion planning, synchronized multimodal acquisition, and 3D reconstruction.The pipeline is summarized in Fig. 1.Source: A. Overview · p. 3
  • A Kinova Gen 2 arm carries RGB-D, thermal, and 2D LiDAR sensors; LiDAR estimates the target volume, while the cameras acquire visual, geometric, and thermal data.After calibration, the system bounds the target, generates and aims viewpoints, plans collision-free motions with MoveIt, and logs synchronized sensor and pose data for reconstruction.Source: A. Overview · p. 3

B. Multimodal Sensor Head

The sensor head combines RGB-D, LiDAR, and thermal sensing for inspection, while Gazebo supports repeatable tests of viewpoint generation and collision-free planning. The simulated setup approximates planar mobility rather than full free-floating spacecraft dynamics.

B. Multimodal Sensor Head

  • The end-effector sensor head uses RGB-D for appearance and depth, LiDAR for a coarse target-volume estimate, and thermal imaging for radiometric inspection cues.The LiDAR-derived bounding box supports autonomous viewpoint generation; LiDAR is not intended as the primary reconstruction sensor.Source: B. Multimodal Sensor Head · p. 3
  • The fixed sensor mounting keeps the sensors in a stable relative configuration so calibration parameters remain valid during acquisition.The support is custom 3D-printed and attaches the sensor head to the Kinova Gen 2 end-effector.Source: B. Multimodal Sensor Head · p. 3
  • Radar and event cameras are excluded; the prototype targets close-range, stop-and-acquire inspection with RGB-D, LiDAR, and thermal sensing.The paper notes radar's longer-range measurements and event cameras' advantages under rapid motion and extreme illumination transients.Source: B. Multimodal Sensor Head · p. 3

C. Gazebo-Based Simulation Environment

  • Gazebo testing reveals self-collision and workspace limitations before execution on hardware.The environment supports repeatable validation of viewpoint generation and collision-free planning.Source: C. Gazebo-Based Simulation Environment · pp. 3–4
  • The simulated manipulator uses a holonomic mobile base to approximate planar relative mobility, not the full dynamics of a free-floating spacecraft.The configuration is intended to match motion restricted to a plane in terrestrial microgravity test benches and future orbital robotics laboratories.Source: C. Gazebo-Based Simulation Environment · p. 4

IV. CALIBRATION AND DATA ACQUISITION

The section establishes camera intrinsics and the end-effector-to-camera transform needed to interpret images and use robot poses for reconstruction. An ArUco marker is used only during calibration; the inspection itself does not rely on markers attached to the structure.

A. Camera Calibration

  • Planar-pattern calibration estimates RGB-camera focal lengths, principal point, and distortion coefficients for image interpretation and reconstruction.Accurate intrinsics are important when reconstruction uses robot-derived poses.Source: A. Camera Calibration · p. 4

B. Eye-in-Hand Calibration

  • Eye-in-Hand calibration estimates the rigid transform between the robot end-effector and the camera optical frame.The formulation uses robot end-effector relative motion A, camera-observed relative motion B, and unknown transform X [16],.Source: B. Eye-in-Hand Calibration · p. 4
  • An ArUco marker supports calibration across several known robot poses, after which it is no longer required for markerless inspection.The estimated transform is incorporated into the ROS 2 transform tree using easy handeye; the marker is not attached to the inspected structure.Source: B. Eye-in-Hand Calibration · p. 4

C. Pose Logging

The pipeline estimates an approximate target volume from LiDAR data to guide viewpoint generation, then logs each acquired image with its camera pose. The pose association supports coherent reconstruction datasets.

C. Pose Logging

  • Each acquisition image is paired with its camera pose from the ROS 2 transform tree, stored as a translation vector and orientation quaternion.The resulting structured dataset associates each image with its spatial pose.Source: C. Pose Logging · p. 4
  • Accurately timestamped, coherent camera poses matter because pose quality strongly influences neural reconstruction quality.The passage names Nerfacto and 3D Gaussian Splatting as examples.Source: C. Pose Logging · p. 4

A. Initial LiDAR-Based Volume Estimation

  • A known-trajectory LiDAR scan is combined with robot poses to produce a 3D point cloud, which is filtered to extract an approximate target bounding box.The filtering removes floor, ceiling, and distant measurements; the box guides viewpoint generation rather than precisely reconstructing the object.Source: A. Initial LiDAR-Based Volume Estimation · p. 4

B. Viewpoint Generation

The system places reachable viewpoints around an estimated target volume and directs the camera toward its centroid. MoveIt plans collision-free travel between viewpoints, with the available coverage depending on the robot setup.

B. Viewpoint Generation

  • Viewpoints are distributed around the estimated bounding box to maximize visual coverage while remaining within the robot’s reachable workspace.The camera orientation points its optical axis toward the box centroid; Fig. 3 depicts the generated viewpoints.Source: B. Viewpoint Generation · p. 4, Figure 3 · p. 4
  • In the physical laboratory setup, the fixed manipulator restricts reachable inspection viewpoints to a frontal sector, while simulation can cover the full perimeter.Viewpoint generation adapts to the inspection scenario.Source: B. Viewpoint Generation · p. 4

C. Motion Planning

  • MoveIt plans motion between viewpoints by solving inverse kinematics and generating trajectories checked for environmental and robot self-collisions.The planner also helps the fixed-base setup respect joint limits, avoid singularities, and find feasible motions.Source: C. Motion Planning · p. 5

D. Synchronized Data Acquisition

The inspection dataset is captured with synchronized sensor data and poses, then processed with a selected reconstruction method. In simulation, the resulting reconstruction supports spatially coherent inspection data and was judged to have sufficient visual coverage.

D. Synchronized Data Acquisition

  • At each viewpoint, the robot stops before capturing RGB, thermal, and depth data with the camera pose, reducing motion blur and tying the pose to a stable configuration.Images are stored with timestamps, translation, orientation, and acquisition parameters in a structure that supports conversion to reconstruction frameworks such as Nerfstudio.Source: D. Synchronized Data Acquisition · p. 5

VI. 3D RECONSTRUCTION METHOD SELECTION

  • The candidate-method benchmark is simplified in Table I and compares criteria relevant to robotic inspection rather than presenting the full quantitative benchmark.The stated criteria are visual quality, geometric consistency, and ease of integration with the robotic dataset.Source: §VI · p. 5, Table I · p. 5
  • Nerfacto was selected for experimental validation for its balance of reconstruction quality, robustness with posed image datasets, and practical integration.The reconstruction is treated as an inspection product for coverage assessment, visual analysis, and future multimodal diagnosis.Source: §VI · p. 5

A. Experiment 1: Simulated Inspection with Holonomic Base

  • The simulated inspection used a Kinova Gen 2 on a holonomic base to scan the target, generate viewpoints from its bounding box, plan collision-free paths, and record RGB-D data with camera poses.This setup enabled movement around the structure and acquisition from a complete set of viewpoints.Source: A. Experiment 1: Simulated Inspection… · p. 5
  • The Nerfacto reconstruction from the simulated experiment showed sufficient visual coverage and spatially coherent stored poses, as illustrated with the input RGB images in Fig. 4.The simulated pipeline also served to test inspection strategies before physical deployment.Source: A. Experiment 1: Simulated Inspection… · p. 5, Figure 4 · p. 5

B. Experiment 2: Physical Validation with Fixed Base

The fixed-base laboratory experiment tested autonomous inspection under restricted reach, validating viewpoint generation, data acquisition, and reconstruction over approximately the frontal sector of the structure.

B. Experiment 2: Physical Validation with Fixed Base

  • Inverse kinematics, singularity avoidance, and motion planning between viewpoints were delegated to MoveIt.The physical setup used a fixed-base Kinova Gen 2 to inspect a vertical structure, with more severe workspace limits than simulation.Source: B. Experiment 2: Physical Validation… · p. 5
  • The fixed-base manipulator estimated the structure volume, generated feasible inspection viewpoints, and executed the acquisition sequence.Reachable viewpoints covered approximately the frontal sector, not the full 360-degree structure.Source: B. Experiment 2: Physical Validation… · pp. 5–6
  • The reconstruction from the physical experiment was spatially coherent enough for visual and geometric assessment under constrained kinematics.The two left images show the real object and the two right images show its reconstruction.Source: Figure 5 · p. 6, B. Experiment 2: Physical Validation… · p. 6

C. Thermal-Aware Reconstruction

Thermal-Nerfacto was evaluated as a way to render synchronized thermal data in alignment with the reconstructed scene. The experiments show thermal contrast alongside RGB context, while the broader validation remains preliminary and does not reproduce orbital conditions.

C. Thermal-Aware Reconstruction

  • Thermal-Nerfacto used synchronized RGB and thermal images to train a reconstruction that renders scene appearance and thermal response.The experiment assessed an inspection-relevant representation, not a thermal point cloud; thermal views may support diagnosis while exported geometry is based primarily on RGB.Source: C. Thermal-Aware Reconstruction · p. 6
  • Fig. 6 shows a thermal rendering that preserves scene contrast, with warmer regions that could support future inspection and anomaly analysis.RGB reconstruction supplies visual and geometric context while the thermal rendering shows surface temperature distribution.Source: Figure 6 · p. 6, C. Thermal-Aware Reconstruction · p. 6

D. Discussion

  • The simulated experiment demonstrated complete multiview acquisition when a mobile base and manipulator could move around the structure.The physical experiment instead validated real sensors, calibration, ROS 2 acquisition, and reconstruction with a fixed base and reduced coverage.Source: D. Discussion · p. 6
  • Accurate calibration and stable data acquisition are central because inconsistencies in camera pose, sensor mounting, or intrinsic parameters can affect reconstruction.Source: D. Discussion · p. 6
  • Thermal rendering represents target temperature distribution for future diagnosis, while exported geometry remains mainly supported by RGB reconstruction.Potential diagnostic regions may be defined by abnormal thermal behaviour as well as geometric defects.Source: D. Discussion · pp. 6–7
  • Validation remains preliminary and system-oriented, without reproducing the complete dynamics, illumination conditions, or operational constraints of an orbital environment.Source: D. Discussion · p. 7

VIII. CONCLUSION

The paper presents a markerless robotic inspection framework that uses 3D reconstruction as support for inspection rather than as its final objective. Complementary simulated and laboratory tests, together with reconstruction experiments, show spatially coherent inspection data, while more representative orbital validation remains future work.

VIII. CONCLUSION

  • The framework combines a Kinova Gen 2, multimodal sensing, ROS 2 acquisition, LiDAR-based volume estimation, viewpoint generation, MoveIt planning, and synchronized RGB-D, thermal, and pose logging.3D reconstruction serves as an inspection-support representation.Source: §VIII · p. 7
  • The architecture was validated in Gazebo with a holonomic base and in a fixed-base laboratory experiment, with an additional thermal-aware reconstruction experiment.The physical test validated sensor integration, calibration, pose logging, and acquisition under available hardware constraints.Source: §VIII · p. 7
  • Nerfacto was selected for its balance of visual quality, geometric coherence, and integration with the robotic dataset; Thermal-Nerfacto demonstrated aligned thermal renderings.The resulting outputs show spatially coherent inspection data.Source: §VIII · p. 7
  • Future work targets representative orbital test-bench validation, stronger thermal-reconstruction integration, and local high-resolution inspection of possible damage or thermal anomalies.Source: §VIII · p. 7
Loading 2609.29644v1…