PaperlayerView exact revision on arXiv

Source-linked AI summary

EBV-CAN: Vision--Language Grounding for Contact-Aided Endoscope Navigation

Chi Kit Ng, Shuchang Liu, Jinsong Lin, Yidong Zhang, Xizi Wang, Ho Yin Chim, Hongliang Ren

arXiv:2610.30786v1cs.RO

TL;DR

Endoscope navigation must relate changing local observations to route progress in a global scene seen only once. EBV-CAN grounds language-conditioned spatial estimates in that fixed view and uses them to predict motor commands. In physical trials it reached 76.25% success, exceeding the strongest baseline by 60.0 percentage points, while action-imitation and completion rankings differed.

  • Problem

    How evolving local endoscopic evidence updates route progress in a global scene observed once remains a central question, because navigation needs persistent spatial context and changing tip and waypoint estimates.

    Source: §I · p. 1

  • Method

    EBV-CAN grounds language-conditioned tip and route-progress estimates in one fixed global image, updating them from evolving endoscopic evidence to predict motor commands.

    Source: §I · p. 2

  • Results

    76.25% success over 240 physical trials exceeded the strongest baseline by 60.0 percentage points, while action-imitation and physical-completion rankings differed, with GRU-BC achieving the lowest imitation error.

    Source: §I · p. 2, §VII · p. 11

  • Takeaways & Limitations

    Action fidelity alone gives an incomplete ranking of navigation performance, supporting evaluation of action fidelity, spatial progress, and physical completion together.

    Source: §VII · p. 11, §VIII · p. 12

    Validation covers planar, rigid guide structures, stable scene registration, two control DoFs, and shared layout families; deformable three-dimensional anatomy remains a distinct validation stage.

    Source: §VII · p. 11

Abstract

from arXiv · show

Flexible endoscopes exploit structural compliance to traverse narrow lumens, pipes, and cavities, where navigation involves distributed environmental contact. This compliance makes tip motion depend on the evolving support configuration. We present external bird's-eye-view contact-aided navigation (EBV-CAN), a vision-language policy that combines one initial global static bird's-eye image with evolving endoscopic observations and motor state. Three supervised semantic queries locate the tip, next waypoint, and instructed target on the cached global image; their residual representations predict bending and insertion commands. We introduce the Vision-Language Contact-Aided (VLCA) dataset, linking 517 demonstrations and 75k action samples to 958k spatial entries and 93k remaining-route records. On 50 held-out episodes, tip-coordinate and waypoint localization reach 97.5% and 94.8%. Swapping the target instruction under identical observations changes target attention and motor output across 17 dual-target episodes. Across 1,920 physical trials of eight policies trained under a shared protocol, EBV-CAN achieves 183/240 successes (76.25%) under automatic blue-area scoring. Gated recurrent unit behavior cloning (GRU-BC) achieves the lowest matched-task offline mean absolute error (MAE), 0.056, and 1/240 physical successes, establishing distinct rankings for action imitation and task completion. Successful multi-guide executions and representative failures connect spatial progress, steering transitions, and visibility to physical navigation.

I. INTRODUCTION · II. RELATED WORK · A. Contact-Aided Continuum-Robot Motion

The paper frames contact-aided navigation as a sequential, locally observed control problem with changing shaft support and a persistent global scene, then connects spatially grounded language-conditioned predictions to motor commands. It introduces VLCA and reports that physical task completion and offline action imitation rank policies differently.

I. INTRODUCTION

  • Insertion, friction, and external support change how motor commands move the tip; a guide can support the passive shaft so further insertion and distal bending redirect it toward a target,.The illustrated envelope is a qualitative schematic, and Rao et al. plan contact-supported motions.Source: §I · p. 1, Figure 1 · p. 1
  • Endoscopic views reveal nearby geometry while the initial global view locates guides and targets, motivating persistent spatial reference and changing tip and waypoint estimates –.The paper asks how evolving local evidence updates route progress when the global scene is observed once.Source: §I · p. 1
  • Offline action MAE measures agreement at recorded states, whereas physical execution generates new observations through policy actions, so the paper evaluates action fidelity, spatial progress, and physical endpoints.The paper also reports spatial and instruction interventions examining how local feedback and target language affect predictions.Source: §I · p. 2
  • The policy uses evolving endoscopic evidence and language-conditioned tip, waypoint, and target queries on one initial static global image to predict bending and insertion commands.The query residual representations connect spatial supervision for route progress to feedback control.Source: §I · p. 2
  • VLCA links 517 physical demonstrations, 74,693 action samples, and 957,616 spatial entries to object boxes, guide identities, ordered plans, and remaining-route labels.The dataset supports joint study of localization, route transitions, and action prediction.Source: §I · p. 2
  • EBV-CAN succeeds in 76.25% of 240 physical trials, exceeding the strongest baseline by 60.0 percentage points under a shared protocol and automatic endpoint criterion.The paper reports that action imitation and contact-aided task completion yield distinct policy rankings.Source: §I · p. 2

A. Contact-Aided Continuum-Robot Motion

  • Prior work studies contact localization, force sensing, deformation estimation,, compliant structures,, and contact-aided planning or simulation learning,.This paper contributes demonstration-based visual control in physical execution, using an initial scene for guide and target locations and local observations for spatial predictions and motor output.Source: A. Contact-Aided Continuum-Robot Motion · p. 2

B. Visual Feedback for Endoscopic Navigation … C. Spatial Annotation and Route Progress

EBV-CAN links an initial global bird’s-eye view, evolving endoscopic observations, motor state, and spatial route supervision for navigation. The VLCA dataset supplies linked action samples, spatial annotations, and remaining-route records across varied guide, insertion, and target tasks.

B. Visual Feedback for Endoscopic Navigation

  • Visual navigation has used adaptive control, depth-guided planning, virtual eye-in-hand training, and shared-control co-pilots; EBV-CAN pairs an initial external image with ongoing tip-camera observations.Spatial labels connect the views, and episode-disjoint evaluation tests new demonstrations from the recorded layout distribution; success measures remain specific to each platform’s protocol.Source: B. Visual Feedback for Endoscopic… · p. 2

C. Language-Conditioned Learning and Spatial Supervision

  • EBV-CAN combines a fixed EBV reference, changing EV history, spatial route objectives, and star-or-circle language templates, testing target-query changes with motor outputs and visual-input interventions.The evaluated context includes GRU-BC, ACT [26], Diffusion Policy [27], multi-view representations, and Qwen3-VL, GR00T, and StarVLA [30]–[32], with frozen and LoRA [33] configurations.Source: C. Language-Conditioned Learning and… · pp. 2–3

A. Platform, Observations, and Task Design

  • The platform uses motorized distal bending and roller-driven insertion or retraction, with a passive proximal shaft, overhead camera, illuminated tip camera, and Xbox-controller teleoperation.The overhead view records the global scene while the tip camera supplies the EV stream.Source: A. Platform, Observations, and Task… · p. 3, Figure 2 · p. 4
  • Nine task levels vary guide use, insertion-dependent traversal, and target selection; L4 and L4.5 include both star and circle targets, and episodes use one target-conditioned instruction.The two templates select star or circle and share a directive to use guides; ordered route annotations store the episode-specific guide sequence.Source: A. Platform, Observations, and Task… · p. 3

B. Recording and Policy Samples

  • Later EBV images support annotation and visualization, whereas deployment uses the initial EBV image throughout, distinguishing images used for annotation and visualization from the image used during deployment.Source: B. Recording and Policy Samples · p. 3
  • Both cameras record at 640 × 480 pixels, nominally at 20 Hz, while retained observations have nonuniform timing and an eight-observation window spans a median 0.351 s.The EV stream is center-cropped, circularly masked, and resized per encoder; motor positions and preceding actions form the state, with the preceding action initialized to zero.Source: B. Recording and Policy Samples · p. 3

C. Spatial Annotation and Route Progress

  • 94,173 frame-level EBV records contain 957,616 spatial entries: 770,788 guide/object boxes, 93,415 target boxes, and 93,413 robot-tip entries.Boxes and tip locations use normalized image coordinates; the separate 93,413-point coordinate field represents the same tip annotations.Source: C. Spatial Annotation and Route Progress · p. 3
  • 93,085 remaining-plan records identify the next waypoint—25,709 guide and 67,376 target records—and 495 changes encode route progression.Canonical guide IDs link geometry to 518 ordered plans; a guide is removed when the annotated tip approaches its box edge within a configured threshold, usually 20 pixels.Source: C. Spatial Annotation and Route Progress · pp. 3–4

D. Partitions and Dataset Contribution … B. Language-Conditioned Spatial Queries

VLCA links demonstrations, action samples, spatial annotations, and route records to support evaluation of action imitation, spatial estimation, and instruction-dependent outputs. EBV-CAN uses a fixed initial overhead image with evolving endoscopic observations and motor state, while language-conditioned queries locate the tip, waypoint, and target for action prediction.

D. Partitions and Dataset Contribution

  • 417 training, 50 validation, and 50 test episodes contain 60,800, 6,614, and 7,279 action samples, respectively.The split is stratified by task level and target identity with seed 42; whole episodes remain within one partition, while layouts are shared across the recorded distribution.Source: D. Partitions and Dataset Contribution · p. 4
  • 495 route updates support event-centered evaluation around waypoint changes, while VLCA links action imitation, spatial estimation, and instruction-dependence questions.Dense global annotations supervise local-to-global estimation, although execution uses one initial overhead image.Source: D. Partitions and Dataset Contribution · pp. 4–5

A. Static Map and Dynamic Navigation Feature

  • Only the first predicted action is applied before the policy predicts again, using visual history and motor state as dynamic observations.The policy conditions on the initial EBV, recent retained endoscopic observations, motor state, and instruction.Source: A. Static Map and Dynamic Navigation… · p. 5
  • The initial image and its frozen features remain fixed throughout an episode, supplying image-derived map tokens with positional encodings.Source: A. Static Map and Dynamic Navigation… · p. 5
  • A width-256 GRU encodes eight recent endoscopic-view class tokens with motor state, then fuses them with a CNN encoding of the current view.The GRU resets for each finite window, with zero-padded visual history at episode start.Source: A. Static Map and Dynamic Navigation… · p. 5

B. Language-Conditioned Spatial Queries

  • Three learned queries for the tip, waypoint, and target read the global map through cross-attention, producing supervised spatial weights and residual tokens for the action head.The queries are conditioned on the instruction and dynamic state; the cross-attention blocks use four heads, and the final spatial weights cover 768 map patches.Source: B. Language-Conditioned Spatial Queries · p. 5

C. Actions and Training Objectives

EBV-CAN predicts chunked motor actions and tip coordinates from learned representations, while training uses action, spatial-attention, coordinate, and counterfactual supervision. The selected model uses the shared training protocol and an epoch-11 checkpoint.

C. Actions and Training Objectives

  • A 512-dimensional MLP with tanh outputs predicts six two-channel actions, while a separate tip-token head predicts normalized coordinates.The MLP maps the concatenated representation [ht; q(2)tgt,t].Source: C. Actions and Training Objectives · p. 5
  • Training combines masked Smooth-L1 action loss, command smoothness, attention and coordinate losses, with target, coordinate, and counterfactual terms weighted 0.75, 1, and 0.5.Tip attention uses cross-entropy to a Gaussian target; waypoint and target losses use attention mass inside expanded box masks, and the coordinate head uses L1 loss.Source: C. Actions and Training Objectives · p. 5
  • For dual-target episodes, the counterfactual loss swaps star and circle descriptions while fixing observations and state, supervising target-query mass at the other target.Target templates and median sizes are built only from training episodes; motor actions follow the recorded instruction’s expert commands.Source: C. Actions and Training Objectives · p. 5
  • The shared AdamW protocol uses a 10^-4 head learning rate, effective batch 64, 10^-4 weight decay, gradient clipping at 1.0, cosine decay, and a 30-epoch cap.Full-validation first-action MAE selects checkpoints with patience five; EBV-CAN analyses use the epoch-11 checkpoint.Source: C. Actions and Training Objectives · p. 5

V. EVALUATION PROTOCOL … C. Physical Trials

The evaluation protocol measures offline action prediction and spatial grounding, tests language-dependent outputs under matched observations, and assesses physical navigation with an automatic blue-area success criterion.

A. Offline Prediction and Spatial Grounding

  • Offline action prediction uses all 7,279 samples from 50 test episodes and measures first-action MAE in normalized channels and controller units.The preceding action is the demonstrated preceding action, and a copy-previous-action reference quantifies temporal persistence under teacher forcing.Source: A. Offline Prediction and Spatial… · p. 5
  • The implemented policy caches initial bird’s-eye-view patch features and conditions three learned queries on endoscopic history, current state, current image, and language.The action head predicts six two-motor commands and executes only the first; an auxiliary coordinate head uses the tip token.Source: Figure 4 · p. 6
  • Tip and waypoint localization use attention-peak positions, while target and guide mass and top-area intersection-over-union assess agreement with annotated regions.Guide-only statistics use 2,082 samples; target statistics use all 7,279.Source: A. Offline Prediction and Spatial… · p. 7

B. Matched-Observation Language Test

  • 17 held-out dual-target episodes are evaluated twice, swapping star/circle instructions while holding images, history, positions, and preceding action fixed.Strict query switching requires target attention to favor the recorded target for the original instruction and the alternate target for the swapped instruction.Source: B. Matched-Observation Language Test · p. 7
  • Language-test command changes are summarized per episode and bootstrapped over 17 episode means with 10,000 resamples to account for within-episode frame correlation.The paired inference test measures output dependence on language through changes in predicted commands.Source: B. Matched-Observation Language Test · p. 7

C. Physical Trials

  • 1,920 physical trials evaluate eight selected checkpoints, with 240 trials per policy across eight levels from L0 to L4.L4.5 is reserved for offline evaluation, and repeated task configurations are clustered by session and layout.Source: C. Physical Trials · p. 7
  • A physical trial succeeds when the largest blue connected component reaches 5,000 pixels in any of the final three endoscopic-view frames.Table V retains 30 trials in every policy–level cell; nominal 95% Wilson intervals summarize observed proportions.Source: C. Physical Trials · p. 7

VI. RESULTS … C. Sensitivity to Visual Inputs

The selected checkpoint achieves strong tip and waypoint localization, while route-stage analysis exposes errors around waypoint changes. Input interventions show complementary roles for cached global and evolving endoscopic vision, with recent visual history adding temporal information.

A. Dynamic Localization on a Fixed Image

  • 97.5% tip-coordinate accuracy and 94.8% waypoint-peak accuracy characterize the selected checkpoint, alongside 95.8% tip-peak accuracy and 0.024 mean normalized tip-peak error.Target attention mass is 1.000 and guide-only mass is 0.893; their top-area IoU values are 0.580 and 0.519, respectively.Source: A. Dynamic Localization on a Fixed Image · p. 7
  • 0.054 mean waypoint Jensen–Shannon divergence across 6,779 observation pairs ten retained observations apart quantifies temporal change in predictions against the fixed global image.In the held-out demonstration, predicted tip and waypoint locations change with EV evidence and motor state while the background remains fixed.Source: A. Dynamic Localization on a Fixed Image · p. 7

B. Localization Across Route Stages

  • 0.014 mean normalized tip error from the coordinate head improves on the patch-grid estimate’s 0.024 mean error across held-out route stages.The coordinate head’s median error is 0.010 and 95th percentile is 0.037; its equally weighted episode mean is 0.014 with a 95% bootstrap interval of [0.012, 0.015].Source: B. Localization Across Route Stages · p. 7, Figure 8 · p. 8
  • 80.9% waypoint accuracy in transition windows contrasts with 95.7% across the remaining samples; the windows comprise five samples before and five starting at each of 46 changes.The changes occur across 33 episodes; transition windows contain 460 samples, or 6.3% of the test set, but account for 88 of 380 waypoint errors (23.2%).Source: B. Localization Across Route Stages · pp. 7–9, Figure 8 · p. 8
  • 89.4% waypoint accuracy over 2,082 guide-directed samples compares with 96.9% over 5,197 final-target samples; equal episode weighting gives 95.3% overall waypoint accuracy.The 95% interval for the episode-weighted estimate is [92.3, 97.6].Source: B. Localization Across Route Stages · pp. 7, 9, Figure 8 · p. 8

C. Sensitivity to Visual Inputs

  • Repeating the current EV feature changes action MAE from 0.094 to 0.104 and tip-peak accuracy from 95.78% to 92.79%; freezing both EV branches lowers tip and waypoint accuracy to 22.01% and 43.78%.Motor state remains available in the frozen-branch intervention; the results assign the cached EBV a spatial-reference role and evolving EV evidence a configuration-updating role.Source: C. Sensitivity to Visual Inputs · pp. 9–10
  • 3.04% waypoint accuracy after cached global patch features are permuted, with position encodings fixed, and 0.101 action MAE show sensitivity to spatial feature assignment.The permutation preserves the visual-feature set but disrupts its spatial assignment; the authors note that navigation skip and residual queries provide additional information paths to the action head.Source: C. Sensitivity to Visual Inputs · p. 10

D. Language Changes Attention and Actions … G. Successful Progression Through Difficult Tasks

EBV-CAN leads the reported physical benchmark, while offline imitation-error rankings differ from task completion. The remaining results show instruction-conditioned query and action changes, action-fidelity and model-size tradeoffs, and successful navigation through difficult tasks.

D. Language Changes Attention and Actions

  • Strict target-query switching occurred in all 2,145 dual-target samples, and changing only the instruction changed first motor commands across 17 episodes.The frame-weighted mean normalized difference was 0.067 and the equally weighted episode mean was 0.068 (bootstrap 95% interval [0.060, 0.075]); mean absolute insertion and bending changes were 1.236 and 10.871 controller units. This tests protocol compliance, not physical target-switch effectiveness, which requires paired rollouts with alternate-target outcome judgments.Source: D. Language Changes Attention and… · p. 10

E. Action Fidelity and Model Size

  • GRU-BC obtained the lowest learned-policy first-action MAE (0.057), followed by EBV-CAN (0.094) and Diffusion-MLP (0.130); copying the preceding expert action yielded 0.051.The preceding-action result quantifies temporal persistence under teacher forcing.Source: E. Action Fidelity and Model Size · p. 10
  • EBV-CAN uses 4.80 million trainable parameters and approximately 26.9 million deployed parameters, with task learning concentrated in navigation and semantic-query modules.Its frozen visual encoder contributes to the deployed parameter count, and feedback is incorporated after each executed command.Source: E. Action Fidelity and Model Size · p. 10

F. Physical Performance and Offline Rankings

  • 183/240 successes (76.25%) made EBV-CAN the highest-performing policy at every physical task level, 60.0 percentage points above the strongest baseline overall.Frozen GR00T achieved 39/240 successes (16.25%) and LoRA 10/240 (4.17%); GRU-BC and Diffusion-MLP achieved 1/240 and 2/240, while ACT-style and both Qwen heads achieved 0/240.Source: F. Physical Performance and Offline… · p. 10
  • GRU-BC had MAE 0.056 with 0.42% physical success, whereas EBV-CAN had MAE 0.092 with 76.25% success, separating lowest imitation error from highest completion.GR00T variants also had lower MAE than EBV-CAN but lower physical completion; the common training cap and validation selector align model selection, while architectures, pretrained representations, and spatial objectives differ.Source: F. Physical Performance and Offline… · p. 11

G. Successful Progression Through Difficult Tasks

  • Successful L3.5 and L4 executions reached target regions despite large local-appearance changes while retaining the initial global image.L3.5 progressed from a nearby guide surface through an open view to the target; L4 runs reached the blue endpoint under star and circle instructions in scenes containing both target identities.Source: G. Successful Progression Through… · p. 11

H. Zero-Success Cells and Representative Failures

The section quantifies zero-success outcomes across policy-task cells and links representative failures to steering, progression, and visibility. EBV-CAN failures persist on harder navigation and target-selection tasks.

H. Zero-Success Cells and Representative Failures

  • 42 of 64 policy-task cells had zero successes, including seven for GRU-BC, eight each for ACT-style and both Qwen heads, six for Diffusion-MLP, and five for LoRA GR00T.Each 0/30 represents 30 attempts below the endpoint criterion; its nominal Wilson 95% interval is 0–11.4%, while aggregate 0/240 gives 0–1.6%.Source: H. Zero-Success Cells and… · p. 11
  • Representative failures span target pass-bys, mistimed or misdirected steering, reverse progression, occlusion, and loss of target view across the evaluated policies.Figure 10 and SI Sec. S8 connect these behaviors to steering direction, timing-dependent support configurations, and the visibility of subsequent feedback.Source: H. Zero-Success Cells and… · p. 11
  • EBV-CAN successes fell from 29/30 at L0/L1 to 15/30 at L4, with remaining failures involving stalled progression, incorrect or delayed bending, and target-view loss.L1 examples link failures to visible steering and progression errors; recovery behaviors that restore informative views and useful support configurations are identified as a need.Source: H. Zero-Success Cells and… · p. 11

VII. DISCUSSION

The discussion connects fixed global context, evolving local observations, spatial grounding, and semantic intent to physical navigation. It distinguishes imitation accuracy from task completion, identifies route-transition and contact-sensing gaps, and outlines recovery-focused extensions.

VII. DISCUSSION

  • 183 physical completions, held-out grounding, and instruction swaps connect fixed global context and evolving observations to spatial feedback and semantic control.Matched-observation swaps changed both target attention and motor output.Source: §VII · p. 11
  • GRU-BC’s lowest MAE and EBV-CAN’s highest completion rate show that action fidelity alone incompletely ranks navigation performance across expert-state and policy-induced trajectories.Teacher-forced prediction visits demonstrated states and supplies the expert’s preceding action, explaining the 0.051 copy-previous reference; GRU-BC had 1/240 physical successes.Source: §VII · p. 11
  • Transition windows contain 6.3% of samples but 23.2% of waypoint errors, while linked object boxes and remaining-route labels support learning sequential support-transition timing.VLCA uses canonical IDs, ordered plans, and 495 remaining-route updates; boxes locate support geometry, and route labels identify the next relevant object.Source: §VII · p. 11
  • Physical execution still exhibits occlusion, stalled progression, target pass-by, and steering errors, while validation remains limited to planar rigid guides and two control DoFs.Current sensing uses images and motor signals; force and shaft-shape measurements are proposed to resolve contact mechanics and support safety-aware clinical translation, and deformable 3D anatomy requires separate validation.Source: §VII · p. 11
  • Future work will target recovery training and route-transition timing, with visibility-aware retreat and view reacquisition, force and shaft-shape sensing, and deformable 3D routing validation.An RSI loop informed by Endoscopic Recursive Self-Improvement (EndoRSI) [35] will use physical failures to revise demonstrations, objectives, and policies, then evaluate each revision on held-out physical trials.Source: §VII · p. 11

VIII. CONCLUSION

EBV-CAN combines persistent global visual context with evolving local observations and motor state for language-conditioned contact-aided endoscope navigation. Across shared-protocol training and 1,920 physical trials, it achieves 76.25% success, while the lowest-MAE policy achieves 0.42%.

VIII. CONCLUSION

  • EBV-CAN pairs an initial global image for persistent context with local observations and motor state to update spatial predictions and actions.VLCA links 517 demonstrations and 74,693 action samples to dense spatial and remaining-route supervision.Source: §VIII · pp. 11–12
  • 76.25% success for EBV-CAN versus 0.42% for the lowest-MAE policy across 1,920 physical trials shows distinct offline and physical rankings.The conclusion argues for evaluating action fidelity, spatial progress, and physical completion together.Source: §VIII · p. 12
  • Representative failures connect unsuccessful execution to steering, visibility, and recovery challenges, motivating evaluation of action fidelity, spatial progress, and physical completion together.The evaluation draws on shared-protocol training and 1,920 physical trials.Source: §VIII · p. 12

Supplementary Information

The Supplementary Information documents the VLCA platform, dataset and annotation exports, EBV-CAN configuration, and evaluation procedures and outcomes. It also details matched-checkpoint offline and physical comparisons, the shared training protocol and model recipes, navigation-level explanations of zero-success cells, representative failures, and proposed follow-up experiments.

S1. PLATFORM AND TASK FAMILIES

  • The platform combines a one-degree-of-freedom bending knob and insertion rollers with overhead EBV and tip-mounted EV cameras, while policy input uses one frozen initial EBV and evolving EV observations.Nine task families vary route, insertion, and target-selection requirements; both motor channels are available, and planned guides are identified among visible blocks.Source: S1. PLATFORM AND TASK FAMILIES · p. 13

S2.1. Corpus, Filtering, and Partitions; S2.3. Annotation Workflow

  • The retained policy corpus contains 517 episodes and 74,693 action samples, while the broader annotation export has 957,616 spatial entries and 93,085 frame-plan records.Remaining-plan records identify a guide as the next waypoint in 25,709 cases and the target in 67,376; guide progress is labeled by tip-to-box-edge distance against a normalized threshold.Source: S2.1. Corpus, Filtering, and Partitions · p. 13, S2.3. Annotation Workflow · pp. 13–14

S2.6. Additional Structured-Language Annotation Exports

  • Two structured-language exports pair EBV/EV observations with route and grounding annotations, covering 518 recordings in 415 training and 103 evaluation episodes.The plan format uses overlay-enhanced EBV and raw EV; the wild format uses raw images, and both preserve target descriptions, remaining plans, guide-proximity fields, and acceptable next waypoints.Source: S2.6. Additional Structured-Language… · p. 16, Table S2 · p. 16

S3. MODEL AND OPTIMIZATION DETAILS; S4.1. Selected Model; S4.4. Baseline Interpretation

  • Full-validation first-action MAE selected the EBV-CAN checkpoint at epoch 11 (epochs start at one); its test first-action MAE was 0.094, with insertion and bending errors of 1.945 and 13.547 controller units.Source: S4.1. Selected Model · p. 17

S2.5. Derived Alternate-Target Boxes; S4.2. Metric Definitions and Per-Level Results

  • All 2,145 dual-target samples satisfy the derived-reference criterion; with all other model inputs fixed, including the preceding expert action, instruction swaps yield a predicted first-action response with frame-mean normalized L1 difference 0.067 and median 0.054.Equal episode weighting gives 0.068; a percentile bootstrap over 17 episode means (10,000 resamples) gives [0.060, 0.075], describing the predicted first-action response to the instruction swap.Source: S4.2. Metric Definitions and Per-Level… · p. 17

S4.3. Inference-Only Input Interventions

  • 29.69% tip and 3.04% waypoint accuracy under global patch permutation contrast with normalized first-action MAE of 0.101 versus 0.094 with recorded inputs.The permutation preserves patch contents while perturbing their spatial assignment, separating supervised spatial readouts from teacher-forced action error under fixed weights.Source: S4.3. Inference-Only Input Interventions · p. 17

S5.1. Observations, Execution, and Checkpoints; S5.2. Endpoint and Trial Proportions

S5.3. Model-Specific Failure Modes

  • Failures span occlusion, incorrect or mistimed steering, target-view loss, stalled or reversed progression, and difficulty recovering useful visibility across the evaluated policies.Repeated EV appearances despite changing actuator state indicate a recovery problem; the sequences motivate steering-transition supervision and explicit retreat-and-reacquisition behavior, while contact forces and shaft shape remain unmeasured.Source: S5.3. Model-Specific Failure Modes · pp. 18–19

S5.4. Interpretation of Every Zero-Success Cell

  • Across 42 zero-success cells, the supplement distinguishes direct-to-target failures at L0, first-guide steering and visibility failures at L1, and higher-level support-transition and target-task demands.ACT-style and both Qwen heads also have 0/240 aggregate outcomes; frozen GR00T and EBV-CAN have positive counts at every evaluated level.Source: S5.4. Interpretation of Every… · p. 19, Table S6 · p. 20

S5.5. Matched-Checkpoint Offline Comparisons

  • GRU-BC has the lowest eight-level aggregate MAE (0.056) but 1/240 physical successes, while EBV-CAN has MAE 0.092 and 183/240 successes.Frozen GR00T reports 0.061 and 39/240; LoRA reports 0.061 and 10/240. The selected-checkpoint task-level offline MAEs are presented in Table S7.Source: S5.5. Matched-Checkpoint Offline… · p. 19, Table S7 · p. 20

S6. MULTIMEDIA

  • Supplementary Videos 1–7 include teleoperated demonstrations, successful target-conditioned and multi-guide trials, and an unsuccessful L3 trial with incorrect steering.Demonstration views are sampled at 12 frames/s; physical clips pair the fixed initial global image with the endoscopic stream resampled at 20 Hz and use the automatic blue-area endpoint criterion.Source: S6. MULTIMEDIA · p. 19

S7.1. Shared Training Protocol

  • The shared protocol gives policies identical 417/50/50 episode partitions, up to 30 epochs, and checkpoint selection by full-validation first-action normalized MAE.Each epoch visits 60,800 training rows, with 950 optimizer updates; early stopping follows five validation checks with improvement below 10^-4.Source: S7.1. Shared Training Protocol · pp. 19–20

S7.2. Model-Specific Training Recipes

  • Model recipes span finite-history GRU, chunked transformer and diffusion policies, frozen Qwen feature heads, GR00T flow-matching variants, and EBV-CAN target templates.The recipes specify architecture-specific losses and inference procedures; EBV-CAN alternate-target templates use crops from training episodes only.Source: S7.2. Model-Specific Training Recipes · pp. 20–21

S7.3. Optimization Results and Interpretation

S8.1. Evidence for Distinct Failure Behaviors

  • Representative failures include continued insertion under occlusion, mistimed or insufficient steering, saturated or reverse commands, and stalled EBV-CAN progression with limited insertion.Examples include GRU-BC insertion advancing 48 controller units during occlusion, Qwen chunk-head negative insertion in 73.6% of commands, and EBV-CAN net insertion change of 2 units over a stalled 22.05 s sequence.Source: S8.1. Evidence for Distinct Failure… · p. 22

S8.2. Interpretation and Future Experiments

  • Proposed experiments target guide approach, occlusion, target pass-by, and stalled progression with corrective demonstrations, transition-centered objectives, recovery behaviors, and synchronized force and shaft-shape measurements.Paired target-instruction trials and deformable three-dimensional routes would extend the geometric and semantic evaluation.Source: S8.2. Interpretation and Future… · p. 22

(a) GRU-BC

The passages describe time-stamped visual changes during navigation and label steering or progression errors in several policy snapshots.

(a) GRU-BC

  • The Diffusion-MLP snapshot is labeled “Incorrect steering,” while the ACT-style snapshot is labeled “Late steering.”Source: (a) GRU-BC · p. 24
  • At 6.0 s, the guide occupies increasing image area; at 12.0 s, positive insertion persists as the view closes.At 14.2 s, the approach ends at a nearby guide surface.Source: (a) GRU-BC · p. 24
  • The Qwen3-VL + chunk MLP snapshot is labeled “Reverse progression.”Source: (a) GRU-BC · p. 24

(d) Qwen3-VL + MLP

The section records view loss alongside large bending commands for Qwen3-VL + MLP and insufficient steering for frozen GR00T; selected L1 failures are shown across all eight policies.

(d) Qwen3-VL + MLP

  • At 6.7 s, large bending commands accompany view loss in Qwen3-VL + MLP.Source: (d) Qwen3-VL + MLP · p. 24
  • Frozen GR00T shows insufficient steering.Source: (d) Qwen3-VL + MLP · p. 24
  • Fig. S6 shows original EV sequences from selected L1 failures across all eight policies, with identical image margins cropped to emphasize task geometry.Source: Figure S6 · p. 24
Loading 2610.30786v1…