PaperlayerView exact revision on arXiv

Source-linked AI summary

CoralPlan: Observation Skill Selection and Execution for Underwater Robotic Inspection

Yuer Gao, Yu Zhao, Yi Cai

arXiv:2609.31211v1cs.RO

TL;DR

Underwater inspection requires acquiring views suited to the task, but recognising a coral target alone does not determine how the robot should observe it. CoralPlan selects an observation skill from a camera image and task text, then executes it through a shared target-relative motion interface. In the clear-water pool study, hardware observation completion is 77.8% and joint success is 63.9%, with experiments identifying both selection mismatches and incomplete observations after matching selections.

  • Problem

    For complex coral colonies, inspection requires selecting and executing viewing motions suited to the operator’s goal, since viewpoint, standoff, and viewing extent determine acquired features.

    Source: §I · p. 1

  • Method

    CoralPlan uses a vision–language model to select an observation primitive from a current camera image and task text, then maps supported primitives to target-relative motion.

    Source: §I · p. 1, A. From a Camera Frame to the Next… · p. 5

  • Results

    Hardware observation completion reaches 77.8% and joint success reaches 63.9% in the clear-water pool study, and both cohorts contain completed reference-mismatched behaviours.

    Source: §I · p. 2, §VIII · p. 7

  • Takeaways & Limitations

    The interface and experiments make the transition from observation-skill selection to physical acquisition measurable for task-directed underwater inspection.

    Source: §VIII · p. 8

    Execution evaluates one adapted planner checkpoint, and the pool cohort tests known target instances; additional geometries, visually estimated target poses, and field trials would extend the evidence.

    Source: §VII · p. 7

Abstract

from arXiv · show

Underwater robotic inspection depends on acquiring views that reveal task-relevant structure. For a structurally complex coral colony, recognising the target is only the starting point: the robot must select and execute a viewing motion suited to the inspection task. We present CoralPlan, a vision-language system that selects an observation skill from a current camera image and task text supplied by an episode manifest. A shared motion interface executes orbit, patch, or survey as target-relative trajectories; the remaining plan fields provide operator guidance. Observation completion requires target keeping and primitive-specific coverage, while joint success also requires selection to match the recorded reference. We evaluate this interface in 144 simulated episodes and 36 matched simulation-hardware pairs. In a clear-water pool with external target-reference poses, hardware observation completion reaches 77.8% and joint success reaches 63.9%. The experiments identify both reference-mismatched completions and incomplete observations after a matching skill selection. These results connect observation-skill choice to measurable underwater execution outcomes and identify where task-directed acquisition succeeds or fails.

I. INTRODUCTION

CoralPlan treats underwater inspection as a choice of how to acquire task-relevant views, not just how to recognise a coral target. It links image-and-task-based skill selection to target-relative motion and evaluates whether the resulting observation meets task-specific criteria.

I. INTRODUCTION

  • Useful coral imagery depends on motion because viewpoint, standoff, and viewing extent determine which features are acquired.Complex three-dimensional forms create occlusions, while close-range checks, multi-angle examinations, and wider surveys require different camera–target geometries.Source: §I · p. 1
  • CoralPlan selects an observation strategy from visual context, executes it, and evaluates the acquired viewing geometry, addressing the target-level interface between perception and physical action.The experiments span offline image–instruction tests, 144 simulated episodes, and 36 matched simulation–hardware pairs.Source: §I · p. 1
  • The shared mapper instantiates orbit, patch, and survey as target-relative trajectories, while view, priority, and task type guide the operator.Only the primitive field configures motion; the remaining plan fields are operator guidance.Source: §I · pp. 1–2, Figure 2 · p. 2
  • Observation completion requires target keeping and primitive-specific coverage, while joint success also requires the selected skill to match the recorded reference.The same definitions are used in simulation and on hardware.Source: §I · p. 2
  • The clear-water pool study uses four known printed coral models and external target-reference poses, while 87 complete wild-reef records are reserved for offline evaluation.Separately captured photographs of the four physical models are included in training.Source: §I · p. 2

II. RELATED WORK

Prior work studies coral-image interpretation, task-directed sensing, and language-guided planning; CoralPlan focuses on selecting and evaluating target-level observation behaviours. CoralVLM provides image–instruction–plan records for the system.

II. RELATED WORK

  • CoralPlan addresses which observation behaviour to execute and whether execution meets viewing requirements, complementing reef-monitoring work focused on what can be inferred from acquired imagery.Prior coral datasets cover benthic annotation, condition recognition, and scene segmentation,, [5].Source: §II · p. 2
  • CoralPlan connects target-level inspection instructions to non-contact observation strategies and evaluates target keeping and acquired coverage.Existing underwater systems map commands to mission-level navigation and survey plans [9],; CORAL combines VLM guidance with dynamics-based waypoint planning [12].Source: §II · p. 2
  • CoralPlan links operator-conditioned strategy selection to execution evaluation within a shared observation interface, complementing viewpoint planners optimising exploration, uncertainty, or surface-coverage gain,,.The paper notes that viewpoint optimisers could serve as lower-level executors within this interface.Source: §II · p. 2

III. CORALVLM: DATA FOR OBSERVATION PLANNING

  • CoralVLM supplies image–instruction–plan records for training and evaluating CoralPlan, together with auxiliary coral-health annotations.The records support adaptation and offline controls.Source: §III · p. 2

A. Record Definition

CoralVLM records pair underwater images and health labels with operator instructions and structured plans. The dataset supports observation planning across distinct primitive geometries, with synthetic and held-out wild evaluation records.

A. Record Definition

  • Each CoralVLM record contains an underwater image, a health label, an operator instruction, and a JSON plan.Health labels define stratification for the balanced synthetic subset.Source: A. Record Definition · p. 2

B. Inspection Primitives

  • Inspection primitives link goals to camera–target geometry and trajectory families: patch is a close-range hover, orbit a multi-angle standoff, survey a wide-area transect, and top-view nadir station-keeping.The motion interface supports orbit, patch, and survey; top-view is evaluated only at the planning level.Source: B. Inspection Primitives · p. 2, Figure 3 · p. 3

C. Annotation Pipeline and Statistics

  • The released set contains 2,478 synthetic records and 100 wild-reef images, of which 87 complete records form the held-out wild planning set.The balanced synthetic subset has 1,792 records split into 1,432 training, 180 validation, and 180 synthetic-test records; separately captured photos of four physical coral models are used in training.Source: C. Annotation Pipeline and Statistics · p. 3
  • The synthetic test split is 31.1% survey, 26.1% top-view, 25.6% orbit, and 17.2% patch, with survey as the majority reference primitive.Balancing by health class does not balance primitive labels.Source: C. Annotation Pipeline and Statistics · p. 3
  • Across 290,160 train–test and validation–test image pairs, the minimum synthetic-set dHash distance is 6, and none meets the near-duplicate threshold of ≤5.The comparison covers 257,760 train–test pairs and 32,400 validation–test pairs.Source: C. Annotation Pipeline and Statistics · p. 3

IV. CORALPLAN: FROM INTENT TO OBSERVATION PLANS

CoralPlan predicts a structured observation plan from an image and task text, with the selected primitive determining motion and other fields guiding the operator. The model is adapted to the task, and output parsing and reference adaptation determine which predictions can be executed.

IV. CORALPLAN: FROM INTENT TO OBSERVATION PLANS

  • CoralPlan uses a vision–language model to select an observation plan, and a shared mapper turns supported primitives into motion.The plan’s primitive is the motion-driving field; other fields provide operator guidance.Source: §IV · p. 3, A. Task Formulation · p. 3

A. Task Formulation

  • At deployment, CoralPlan receives the current camera frame and task text from the episode manifest, with matched simulation and hardware episodes using the same task text.The plan includes primitive, view, priority, and task type slots.Source: A. Task Formulation · p. 3

B. Model and Adaptation

  • CoralPlan adapts Qwen2.5-VL-7B [18] with LoRA, freezing the vision encoder; the seed-42 checkpoint is used for robot execution trials.The image-only reference uses the same adaptation recipe without the instruction; additional seeds are evaluated offline.Source: B. Model and Adaptation · p. 3

C. Output Parsing and Reference Adaptation

  • The parser accepts predictions with at least three recovered fields and normalises values with a fixed synonym map, but execution requires a supported primitive.Unsupported or unparseable predictions count as non-executable failures whenever execution is required.Source: C. Output Parsing and Reference… · pp. 3–4

V. FROM OBSERVATION SEMANTICS TO MOTION

CoralPlan turns a camera image and manifest task text into target-relative motion through a shared interface for orbit, patch, and survey. The execution trial matrix defines how target keeping and primitive-specific coverage are evaluated.

V. FROM OBSERVATION SEMANTICS TO MOTION

  • Each episode links the planner’s selected primitive, its instantiated trajectory, and controller execution to newly acquired views and evaluation of selection and observation.Evaluation checks selection against the recorded reference and measures target keeping and coverage.Source: §V · p. 4

A. Execution Environment

  • The simulated environment matches the physical ROV’s geometry, propulsion, camera placement and field of view, and pool workspace.Both platforms use the same primitive-to-motion interface, but select primitives separately from their own images.Source: A. Execution Environment · p. 4

B. Predicted-Plan Execution Loop

  • A current camera frame and manifest task text drive a structured plan whose orbit, patch, or survey primitive is instantiated as a target-relative trajectory.The mapper supports a circular standoff path, close-range hover, or wide-area transect; other plan fields provide operator guidance.Source: B. Predicted-Plan Execution Loop · p. 4

C. Execution Trial Matrix

  • The 144-episode factorial suite varies four coral targets, four turbidity levels, three lighting levels, and three reference inspection primitives.Episodes are grouped by reference primitive; supported predictions are executed even when they differ from the reference.Source: C. Execution Trial Matrix · p. 4
  • Observation completion requires target keeping and the executed primitive’s coverage criterion, with target keeping defined as K ≥0.8.Empty or missing target masks count as failures; only corrupted or unsynchronized frames are excluded from the denominator.Source: C. Execution Trial Matrix · p. 4
  • Orbit coverage requires at least 150° of target-relative azimuth span, while patch coverage requires the camera to remain near a 0.30 m standoff for 80% of evaluated frames.Patch distance tolerance is ±0.05 m.Source: C. Execution Trial Matrix · p. 4
  • Survey coverage requires more than 3.50 m of target-relative position span projected onto the commanded survey axis.Using span rather than travelled distance prevents oscillation from inflating coverage.Source: C. Execution Trial Matrix · p. 4

D. Paired Physical Evaluation

The paired evaluation compares 36 clear-water hardware episodes with matched simulations, with each platform selecting and executing its own prediction. Outcomes are scored separately for selection agreement, observation completion, and joint success.

D. Paired Physical Evaluation

  • The paired cohort contains 36 hardware episodes across four targets, three reference primitives, and three repetitions, each linked to a simulated counterpart.Both platforms execute their own predictions, and pairs with different actions remain in the evaluation.Source: D. Paired Physical Evaluation · p. 4
  • Hardware uses manually verified image masks for observed targets and registered target geometry projected with synchronized motion-capture pose for reference masks.Masks are used only for offline scoring; motion capture also supplies target-reference poses for waypoint instantiation and evaluation.Source: D. Paired Physical Evaluation · p. 4
  • Table I reports selection and observation outcomes as percentages with counts, and joint success requires both reference-matched selection and observation completion.Coverage is scored for the primitive executed on each platform.Source: Table I · p. 5, D. Paired Physical Evaluation · p. 5

VI. EXPERIMENTS

The experiments examine CoralPlan’s path from planning to physical outcomes through offline modality controls, factorial simulation, and paired deployment. They first examine skill selection and then the observations obtained through execution.

VI. EXPERIMENTS

  • Offline planning evaluates model predictions on image–instruction records to measure modality use.The controls compare instruction and matched-image contributions.Source: A. From a Camera Frame to the Next… · p. 5
  • Factorial simulation executes predictions across 144 combinations of target, turbidity, lighting, and reference primitive.The results examine selection before the observations obtained through execution.Source: A. From a Camera Frame to the Next… · p. 5
  • Paired deployment compares 36 physical trials with simulated counterparts, with each platform selecting from its own camera image.The experiments examine selection first, then the observations obtained through execution.Source: A. From a Camera Frame to the Next… · p. 5

B. Offline Planning Measures Instruction and Image Contributions

Offline tests compare instruction and image contributions to skill selection; simulation results show that selection agreement and observation completion do not always coincide.

B. Offline Planning Measures Instruction and Image Contributions

  • 74.4 ± 1.1% synthetic-test primitive accuracy exceeds majority prediction at 31.1%, keyword rules at 54.4%, instruction-only adaptation at 64.8 ± 1.4%, and the image-only reference at 42.8 ± 1.1%.Results are across three training seeds.Source: B. Offline Planning Measures… · p. 5
  • 74.4 ± 1.1% primitive accuracy and 50.9 ± 0.8% exact-plan accuracy exceed shuffled-image results of 67.3 ± 0.8% and 42.8 ± 0.6%, respectively.The shuffled-image control retains instructions while disrupting image–instruction correspondence.Source: B. Offline Planning Measures… · p. 5
  • On the wild split, majority prediction has the highest primitive accuracy, while CoralPlan has higher aggregate Plan-F1 across the four plan fields.The two metrics reflect different aspects of performance on this class-concentrated split.Source: B. Offline Planning Measures… · p. 5

C. Simulation Exposes the Difficulty of Multi-Angle Observation

  • Only 20/34 (58.8%) reference-matched orbit episodes meet both target-keeping and the 150° sweep requirement, while all matched patch and survey episodes complete.The paper identifies multi-angle observation as the most difficult of the three behaviours under the main criteria.Source: C. Simulation Exposes the Difficulty… · p. 5
  • 111/144 (77.1%) selections match the recorded reference, 121/144 (84.0%) episodes complete observation, and 97/144 (67.4%) achieve both.These are results across the full 144-episode factorial suite.Source: C. Simulation Exposes the Difficulty… · p. 5
  • The 24-episode gap between observation completion and joint success comprises reference-mismatched completions, while 14 reference-matched episodes fail observation completion.Nineteen of the 24 mismatched completions occur in patch and survey reference groups.Source: C. Simulation Exposes the Difficulty… · p. 6
  • Orbit completion ranges from 6–7/12 across turbidity levels and 7–9/16 across lighting levels, versus 0/12–12/12 across the four tested targets.The passage highlights variation among the tested colonies.Source: C. Simulation Exposes the Difficulty… · p. 6
  • Completion falls from 121/144 to 112/144, 100/144, and 71/144 under progressively tighter joint criteria; orbit-group completion remains 25/48.The rescore uses frozen seed-42 episodes without replanning or rerunning trajectories; five non-orbit executions in the orbit reference group remain complete under strict criteria.Source: C. Simulation Exposes the Difficulty… · p. 6

D. Hardware Demonstrates Selection and Observation Outcomes

In the 36-pair pool evaluation, hardware reached 77.8% observation completion and 63.9% joint success. Outcomes also distinguish reference-mismatched completions from failures to meet observation criteria after a matching selection.

D. Hardware Demonstrates Selection and Observation Outcomes

  • Five hardware trials completed a behavior different from the reference, while five matched selections failed observation criteria.Of the remaining 26 trials, 23 satisfied both conditions and three satisfied neither; the mismatched completions account for the 13.9 pp gap between hardware observation completion and joint success.Source: D. Hardware Demonstrates Selection and… · p. 6
  • Hardware observation completion was 28/36 (77.8%) and joint success was 23/36 (63.9%), versus 30/36 (83.3%) and 25/36 (69.4%) in simulation.Selection agreement was 28/36 (77.8%) on hardware and 29/36 (80.6%) in simulation.Source: D. Hardware Demonstrates Selection and… · p. 6
  • Predicted and executed primitives each agreed across platforms in 32/36 pairs (88.9%); observation verdicts agreed in 34/36 pairs (94.4%).All four pairs with different executed actions had matching observation verdicts; joint verdicts agreed in 32/36 pairs (88.9%).Source: D. Hardware Demonstrates Selection and… · pp. 6–7
  • Target keeping passed in 35/36 simulated and 34/36 hardware episodes, but five and six episodes respectively retained the target while failing coverage.The keeping/coverage counts were (30, 5, 1, 0) in simulation and (28, 6, 0, 2) on hardware, ordered as both pass, keeping only, coverage only, neither; one simulated episode instead passed coverage but failed keeping.Source: D. Hardware Demonstrates Selection and… · p. 7

VII. LIMITATIONS AND OUTLOOK

The evaluation has limits in grounding evidence, execution coverage, and what its observation criteria measure. The authors identify additional annotations, trials, and downstream image-quality measures as ways to extend the evidence.

VII. LIMITATIONS AND OUTLOOK

  • Generated instructions and plans share a primitive reference, so independent expert annotations and counterfactual input pairs would strengthen grounding assessment.The 87 complete wild-reef records also have a concentrated primitive distribution.Source: §VII · p. 7
  • Execution evaluates only the seed-42 adapted planner; keyword-rule and instruction-only controls are evaluated offline, while additional geometries, visually estimated poses, and field trials would extend the evidence.Orbit-threshold sensitivity and target-specific failure analysis are also identified as further work.Source: §VII · p. 7
  • Current criteria measure observation geometry, not downstream reconstruction quality or diagnostic detail.Evaluating those outcomes would connect skill execution to the utility of acquired imagery for reconstruction and adaptive scanning.Source: §VII · p. 7

VIII. CONCLUSION

CoralPlan links a current underwater view and task text to an executable observation skill, then evaluates target keeping and coverage. The pool study shows both reference-mismatched completions and incomplete observations after matching selections.

VIII. CONCLUSION

  • A shared mapper turns the selected orbit, patch, or survey primitive into target-relative motion, assessed with target-keeping and coverage criteria.In the pool study, hardware observation completion was 77.8% and joint success was 63.9%.Source: §VIII · p. 7
  • In simulation, all 14 reference-matched selections with incomplete observations selected and executed orbit.Reference-mismatched completions account for the gap between observation completion and joint success in both cohorts.Source: §VIII · p. 7
  • The interface and experiments make the transition from observation-skill selection to physical acquisition measurable for task-directed underwater inspection.Source: §VIII · p. 8

Related papers

Loading 2609.31211v1…