Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

I. INTRODUCTION

• Existing instrument-based FoV strategies often follow predefined tool rules, while task-aware methods use phase to guide camera behavior but largely decouple phase from target detection and tracking.The paper identifies occlusion and camera-induced image motion as challenges to reliable target tracking.Source: §I · p. 1
• Phase-dependent relationships between active instruments and manipulated objects, together with the non-active status of other entities, provide cues for maintaining target states under ambiguous observations and occlusion.The framework treats manipulated objects as coupled to active instruments and non-active entities as semantically static.Source: §I · pp. 1–2
• The framework integrates phase recognition with object detection, point tracking, and camera-motion compensation so phase-dependent semantics constrain target state estimation.The recognized phase constrains state updates; non-active entities also support compensation for laparoscope-induced image motion.Source: §I · p. 2
• The refined entity states are organized into a task-aware control representation for FoV control, which is validated in phantom Peg Transfer and Suturing tasks.The framework is presented as coupling existing detection and tracking outputs with task dynamics, rather than replacing those models.Source: §I · p. 2

A. Framework Overview

• Visual Perception produces visual estimates and camera-motion-aware kinematics, Surgical State Inference derives the phase and refines the state, and FoV Control maps that state and phase to a camera command.The refined state feeds back as the next time step's temporal prior, closing the perception-control loop.Source: A. Framework Overview · p. 2, Figure 1 · p. 2

1) Object detection and point tracking:

• Frame-level object detection supplies spatially anchored observations and initializes point tracking, whose propagated points are retained when consistent with reliable detections.The described implementation uses YOLOv8 for detection and TAPNext++ for point propagation.Source: 1) Object detection and point tracking · p. 2
• When detection and tracking substantially disagree, the affected entity is corrected using detection and its tracker is re-initialized.This combines detection and tracking to address their respective limitations.Source: 1) Object detection and point tracking · pp. 2–3
• For each task-relevant entity with a valid observation, the current visual estimate combines detection, tracking, and the previous state; reliability reflects detection confidence and spatial agreement between detection and tracking.The previous state also gates estimates that substantially deviate from the expected state, assigning low reliability or excluding them.Source: 1) Object detection and point tracking · p. 3
• Entities with no reliable observation are maintained for subsequent semantic state estimation rather than discarded.The task-relevant set is specified by the task definition, and the observed subset contains entities with valid observations at the current time.Source: 1) Object detection and point tracking · p. 3

2) Camera-motion-aware kinematics:

• Assuming locally consistent depth, inter-frame camera-induced translation and planar motion are approximated with a 2D homography, primarily for rigid surgical entities in the current scope.The text notes that soft-tissue deformation and respiration occur in vivo.Source: 2) Camera-motion-aware kinematics · p. 3
• The previous state anchors entities expected to be static, and visible regions associated with non-active entities are preferred for homography estimation while task-active or inconsistent regions are excluded.This reduces corruption of the camera transformation estimate by task-related entity motion.Source: 2) Camera-motion-aware kinematics · p. 3
• The estimated camera transformation maps an entity's previous image position into the current image coordinates to produce a camera-motion-compensated prediction, from which task-related image velocity is derived.Fig. 2(a) illustrates the process on a suturing scene.Source: 2) Camera-motion-aware kinematics · p. 3
• Camera-motion-aware kinematics also includes observation-level relative image positions for ordered task-relevant entity pairs, providing relational information alongside individual entity motion.The task definition specifies which entities are paired and their order.Source: 2) Camera-motion-aware kinematics · pp. 3–4

1) Surgical phase recognition:

• The current surgical phase is estimated from a temporal window of kinematics.The window length is denoted by τ.Source: 1) Surgical phase recognition · p. 4
• A task-specific temporal convolutional network is trained on kinematic sequences after camera-motion compensation to recognize surgical phase.The network is denoted by H, and τ denotes the window length.Source: 1) Surgical phase recognition · p. 4
• To provide the network a fixed-dimensional input, the variable entity set is mapped to a predefined fixed-order feature vector, with unobserved entities zero-padded and marked by a binary presence mask.Source: 1) Surgical phase recognition · p. 4

• The initial state is built from first-frame YOLOv8 detections and TAPNext++ tracking points, with phase initialized to a default starting phase such as Idle.Source: 2) Semantic state estimation · p. 4

a) Phase-dependent relational constraints:

• Phase-dependent relational constraints identify active instrument–object pairs, while non-active entities are expected to remain nearly stationary.The relationship representation is task-independent, but each task specifies its phase-to-relationship mapping.Source: 2) Semantic state estimation · p. 4

• The maintained task state combines individual entity states with inter-entity relative positions for semantic state estimation.Source: 2) Semantic state estimation · p. 4

c) State maintenance and recovery:

• During occlusion, non-active entities use camera-motion-compensated predictions, while coupled active entities can be inferred from an observable counterpart and retained relative position.Source: 2) Semantic state estimation · p. 4
• If both a coupled counterpart and an established coupling are unavailable, the active entity is temporarily maintained using TAPNext++ tracking output.Source: 2) Semantic state estimation · p. 4
• After reliable observations return, states are re-anchored; a reappearing target is matched to its original identity only when spatial and appearance thresholds are met.Otherwise, the prior state is discarded.Source: 2) Semantic state estimation · p. 4

• Autonomous FoV control consists of structured control signal encoding and RCM-based FoV control.Source: D. Autonomous FoV Control · p. 5
• The control output is a laparoscope motion command, ˙qt.Source: D. Autonomous FoV Control · p. 5
• The two named components together make up the autonomous FoV control stage.Source: D. Autonomous FoV Control · p. 5

1) Structured control signal encoding:

• The structured signal selects active entities and their refined image positions rather than passing the full maintained state to the controller.The active set can include multiple instruments and manipulated objects.Source: 1) Structured control signal encoding · p. 5

2) RCM-based FoV control:

• A phase-dependent aggregation operator combines positions of all active entities to form the FoV target, with a weighted centroid as one possible choice.Source: 2) RCM-based FoV control · p. 5
• The controller generates laparoscope motion from the target’s image-space centering error while satisfying RCM kinematic constraints.Source: 2) RCM-based FoV control · p. 5
• Executed motion produces the next camera observation, closing the perception-control loop.Source: 2) RCM-based FoV control · p. 5

III. EXPERIMENTS

• Three experiments evaluate component contributions, compare phase recognition with state-of-the-art methods, and validate autonomous FoV control on a physical robotic laparoscope platform.Source: §III · p. 5

1) Configurations:

• The ablation compares detection only, detection plus tracking, those components plus camera compensation, and the full method.Source: 1) Configurations · p. 5

2) Datasets and metrics:

• SAR-RARP50 tests camera-motion compensation, Peg Transfer tests discrimination under block overlap, and JIGSAWS suturing tests semantic state maintenance and re-anchoring.Camera-motion compensation is evaluated only on SAR-RARP50 because the other datasets have negligible laparoscope motion.Source: 2) Datasets and metrics · p. 5
• Tracking is assessed with MPE, IDS, and ORL, while phase recognition uses video-level accuracy and phase-level Jaccard coefficient.Tracking annotations were manually labeled and cross-checked by two annotators.Source: 2) Datasets and metrics · p. 5

3) Results:

• The full method performs best overall on all three datasets, particularly Peg Transfer and JIGSAWS under occlusion and overlap; tracking, compensation, and semantic constraints play complementary roles.The ablation results also report that adding TAPNext++ consistently reduces MPE, IDS, and ORL compared with detection.Source: 3) Results · pp. 5–6

1) Setups:

• Phase-level precision and recall are additionally reported to evaluate surgical phase-recognition accuracy and completeness.Source: B. Experiment II: Comparison with SOTA… · p. 6
• Experiment II compares Surgical State Inference with NFSM, MS-ASCT, STSANet, and LemonFM using the same three datasets and data splits.Table II summarizes the phase-recognition comparison.Source: 1) Setups · p. 6, Table II · p. 6

2) Results:

• On SAR-RARP50, the method leads in AC and JA, matches the best baseline in PR, and is slightly lower in RE; on Peg Transfer, it beats all baselines on all four metrics.The Peg Transfer margin is clear despite frequent entity overlap.Source: 2) Results · p. 6
• On JIGSAWS, the method leads in PR and RE while remaining competitive in AC and JA; Fig. 3 shows closer-to-ground-truth Peg Transfer phase transitions with fewer overlap-related fragments and spurious transitions.Source: 2) Results · p. 6, Figure 3 · p. 6

1) Experimental protocol:

• Four operators, including novices and experts, each perform five trials per task under fixed-FoV, tracking-based, and proposed control, yielding N = 20 trials per condition per task.The condition order is counterbalanced across operators; tracking-based control retains the phase recognizer, target aggregation rule, camera-motion compensation, and RCM controller but bypasses relational state updates and occlusion handling.Source: 1) Experimental protocol · p. 6

2) Hardware setup:

• The phantom platform combines a 7-DoF robotic arm holding a laparoscope, a surgical monitor, and a training box with standard FLS Peg Transfer and Suturing modules.Perception, recognition, and control run on a laptop with a GeForce RTX 5070 Ti Laptop GPU.Source: 2) Hardware setup · p. 6

3) Proposed FoV control strategy:

• The FoV target is the centroid of the phase-specific active-entity set; the event-triggered controller moves only after target departure from the outer region and stops upon return to the central safe region.It enforces joint-velocity saturation, holds position when active entities lack reliable state estimates, and permits immediate operator override.Source: 3) Proposed FoV control strategy · pp. 6–7

4) Metrics:

• Evaluation covers image-space errors, target containment and exits, task completion time, camera path length, control latency, and end-to-end update frequency.Source: 4) Metrics · p. 7

5) Results and Discussion:

• Across both tasks, semantic-coupled control significantly reduces pixel error, increases time inside Ωc, nearly halves exits from Ωout, and shortens completion time versus fixed-FoV control (p < 0.001).Source: Table III · p. 7, 5) Results and Discussion · p. 7
• Tracking-based control produces more Ωout exits than fixed FoV and slower completion in Suturing, as occlusion-related tracking errors can cause erratic camera movements that disrupt manipulation.Source: 5) Results and Discussion · p. 7
• Fig. 5 shows tighter target-position concentration around the image center and lower horizontal and vertical deviations under semantic-coupled control than under fixed FoV.Source: 5) Results and Discussion · p. 7
• Camera path length falls from 203.6 mm to 142.0 mm in Peg Transfer and from 132.5 mm to 79.3 mm in Suturing versus tracking-based control, with latency rising to 75.2 ms and update frequency falling to 39.4 Hz.The paper describes the system as retaining favorable real-time responsiveness despite this computational overhead.Source: 5) Results and Discussion · p. 8

IV. CONCLUSION AND DISCUSSION

• The conclusion reports improved target state estimation and more efficient task execution compared with evaluated baselines.Source: §IV · p. 8
• Performance under soft-tissue deformation and respiration remains to be validated in in vivo environments.Source: §IV · p. 8