Source-linked AI summary
ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot
Jian Hu, Shujing He, Leixin Chang, Zongze Li, Ding Huang, Chaoyang Shi, Chengzhi Hu
TL;DR
Autonomous colonoscopy faces challenges from deformable anatomy, weak-texture and specular visuals, and contact-rich interactions. ColoACT combines RGB, relative depth, and pseudo-elevation with chunked, temporally ensembled control; ex-vivo tests report navigation success across straight, curved, and tortuous segments.
Problem
Autonomous colonoscopic navigation is challenged by deformable anatomy, weak texture and specular highlights, and nonlinear contact with viscoelastic tissue, while existing approaches have limitations in visual robustness or sim-to-real transfer.
Source: §I · p. 1Method
ColoACT combines RGB with estimated relative depth and gradient-based pseudo-elevation in an Action Chunking Transformer policy that predicts and temporally ensembles overlapping action chunks for a self-propelled robot.
Source: §I · p. 1, 1) Multi-Cue Visual Input Information… · p. 3, 2) Predictive Action Chunking for… · p. 4Results
Takeaways & Limitations
Ex-vivo validation demonstrates navigation across straight, curved, and complex tortuous porcine-colon segments.
Source: §I · p. 2, §V · p. 8Validation is ex vivo; force sensing and in-vivo trials remain future work.
Source: §V · p. 8
Abstract
from arXiv · showhide
Autonomous colonoscopic navigation can reduce operator burden and the risk of loop formation or tissue trauma, but remains challenging due to deformable anatomy, weak-texture and specular endoscopic visuals, and contact-rich viscoelastic interactions. Existing methods either rely on geometry-driven pipelines, which are efficient and interpretable yet brittle due to manually engineered features and switching logic, or adopt learning-based policies, whose inferred depth/geometry can become temporally inconsistent or overly smooth under weak texture and specular highlights while simulation-trained variants (e.g., deep reinforcement learning) may further suffer from a sim-to-real gap. We propose ColoACT, an autonomous navigation system that integrates an RGB-D-E based Action Chunking Transformer policy (ColoACT policy) for a compact self-propelled Bevel-Gear-Based Endoscopic Robot (BGER). The ColoACT policy augments RGB with estimated relative depth and a gradient-based pseudo-elevation map to enhance fold-ridge saliency and other high-frequency geometric cues, and enables smooth continuous control of the BGER by predicting overlapping action chunks and fusing them via temporal ensembling. In different \textit{ex-vivo} porcine colons (approximately 60 cm), our system achieves success rates of 85.4\% and 72.5\% in straight and curved segments, respectively, and achieves 70\% success in 90-degree turns and 60\% in double-bend sequences, with feasibility further demonstrated in challenging triple-bend segments. The project page is available at: https://Adamhu1.github.io/ColoACT/.
I. INTRODUCTION
ColoACT combines multi-cue visual input, chunked control, and a compact self-propelled robot to navigate ex-vivo porcine colons, including curved and tortuous segments. The system addresses visual and contact challenges that complicate autonomous colonoscopy.
I. INTRODUCTION
- Colonic deformation, weak texture, specular highlights, and contact-rich interactions challenge smooth, robust navigation; existing geometry- and learning-based approaches have distinct limitations.Geometry-driven methods can be brittle, while learned geometric cues may be temporally inconsistent or overly smooth and simulation-trained variants may face sim-to-real gaps.Source: §I · p. 1
- Success rates were 85.4% in straight and 72.5% in curved segments, 70% in 90-degree turns, and 60% in double-bend paths; triple-bend feasibility was also demonstrated.The ex-vivo porcine colons were approximately 60 cm.Source: §I · p. 1
- ColoACT combines RGB, estimated relative depth, and gradient-based pseudo-elevation with an Action Chunking Transformer policy for smooth, continuous closed-loop control on the BGER.The system is presented as an imitation-learning-based navigation policy for a self-propelled colonoscopic robot tested ex vivo.Source: §I · p. 1
- The compact BGER is designed for stable transmission and reliable visual feedback, while system-level validation covers straight, curved, and complex tortuous segments.The robot is 24 × 20 × 34 mm.Source: §I · pp. 1–2, Figure 1 · p. 2
II. RELATED WORK
Prior geometry-driven colon-navigation methods extract explicit visual features and use deterministic rules to guide motion. Their efficiency and interpretability are balanced by sensitivity to noisy cues, deformation, and assumptions that may not hold in real colons.
A. Geometry-Driven Navigation for Robotic Endoscopy
- Geometry-driven methods estimate lumen direction or center using fitted circles, ellipses, triangles, or detected haustral folds to guide motion.The approaches are described as interpretable and computationally efficient.Source: A. Geometry-Driven Navigation for… · p. 2
- Manually engineered thresholds and rule-based switching can produce discrete behavior and sensitivity to perception noise and contact disturbances, especially with weak texture or specular highlights.The cited methods rely on explicit geometric features and deterministic logic.Source: A. Geometry-Driven Navigation for… · p. 2
- Validation primarily on idealized silicone phantoms limits generalization because real colons deform and vary, so lumen geometry can depart from assumed analytic shapes.The discussion links those departures to difficulty sustaining stable closed-loop navigation.Source: A. Geometry-Driven Navigation for… · p. 2
B. Learning-Based Navigation for Robotic Endoscopy
Learning-based navigation can model nonlinear dynamics, but visual degradation, simulation-to-reality differences, and long-horizon consistency remain challenges. ColoACT combines multi-cue perception with chunked, temporally fused actions for closed-loop control.
B. Learning-Based Navigation for Robotic Endoscopy
- Learned depth and mapping can lose accuracy or temporal consistency under deformation, weak texture, and specularities, while simulation-trained reinforcement learning faces difficulty modeling contact-rich colon interactions.Imitation learning reduces dependence on accurate physics, but naive behavior cloning can suffer covariate shift and step-wise policies can produce jittery maneuvers.Source: B. Learning-Based Navigation for… · p. 2
- Figure 2 presents RGB, relative-depth, and pseudo-elevation inputs alongside temporal-variability heatmaps, a dynamic sequence, and depth/elevation profiles.The caption describes the profiles as confirming spatial peak localization.Source: Figure 2 · p. 3
A. ColoACT
- ColoACT augments monocular RGB with relative depth and gradient-based pseudo-elevation, predicts future action sequences, and fuses overlapping chunks to produce smooth BGER velocity commands.Temporal ensembling filters high-frequency jitter and supports navigational stability.Source: §III · p. 3
1) Multi-Cue Visual Input Information Processing:
The visual input combines RGB with estimated relative depth and a gradient-based pseudo-elevation map to represent fine colon topography. Tests report stable cues across static frames and as the robot approaches a fold.
1) Multi-Cue Visual Input Information Processing:
- Depth Anything V2-Base estimates dense relative depth from single RGB frames, while Scharr gradients of normalized depth form a pseudo-elevation map that emphasizes folds and fine-scale relief.A logarithmic transform compresses strong gradients and amplifies weaker informative boundaries before min–max normalization; the resulting map makes fold ridges more distinguishable than in the depth map.Source: 1) Multi-Cue Visual Input Information… · p. 3
- Across 50 static frames, depth and elevation pixel-wise standard deviations were 0.0071 and 0.0250, respectively, while fold peaks remained localized and geometric structures evolved consistently as the robot approached.The reported dynamic sequence showed clear boundaries without artifacts.Source: 1) Multi-Cue Visual Input Information… · pp. 3–4
2) Predictive Action Chunking for Smooth Endoscopic Navigation:
ColoACT combines multi-cue visual observations with ACT action-chunk prediction and temporal ensembling to produce temporally coherent, continuous control in viscoelastic environments.
2) Predictive Action Chunking for Smooth Endoscopic Navigation:
- ACT predicts 16-step chunks of normalized linear and angular velocities, supporting temporally coherent navigation under non-Markovian dynamics and hysteresis.The policy builds on a CVAE framework for predicting future action sequences.Source: 2) Predictive Action Chunking for… · p. 4
- RGB, relative depth, and pseudo-elevation form the visual input, which a modified ResNet-18 encodes into a feature embedding.The pseudo-elevation cue is intended to preserve high-frequency topographic features such as folds and fine-scale relief.Source: 2) Predictive Action Chunking for… · p. 4
- A CVAE encoder samples a latent style variable conditioned on the observation and demonstrated action chunk, and the decoder uses it with visual context to generate actions.Fixed sinusoidal position queries guide the predicted sequence ordering.Source: 2) Predictive Action Chunking for… · p. 4
- Temporal Ensembling averages overlapping chunk predictions to filter high-frequency policy noise and support kinematic continuity.The policy predicts a 16-step horizon; the number of valid overlapping predictions is typically 16 in steady state.Source: 2) Predictive Action Chunking for… · p. 4
B. ColoACT Platform: System and Mechanical Design
BGER is a compact self-propelled tread robot designed for maneuverability and stable movement and visual feedback for learning-based navigation.
B. ColoACT Platform: System and Mechanical Design
- At 24 × 20 × 34 mm, BGER is a compact self-propelled tread robot designed for maneuverability in the confined colon lumen.Adult colons are approximately 3–8 cm in diameter overall.Source: B. ColoACT Platform: System and… · p. 4
- Biocompatible PDMS treads with concave-convex textures provide traction, driven by two independent coreless DC micro-motors with a 700:1 reduction ratio.Source: B. ColoACT Platform: System and… · p. 4
- The Bevel-Gear-Based mechanism vectorially cancels radial and tangential reaction forces, isolating motor shafts from bending loads and reducing mechanical jitter.Source: B. ColoACT Platform: System and… · p. 4
- A 3.3 mm distal monocular vision module provides an 85° field of view and video up to 720p at 30 fps, while a tether and microcontroller support video transfer and motor control.An adjustable LED provides intraluminal illumination, and an emergency-stop button disables motor actuation.Source: B. ColoACT Platform: System and… · p. 4
IV. EXPERIMENT
The experiments evaluate whether ColoACT supports robust, smooth closed-loop BGER navigation in visually degraded, contact-rich colonic environments.
IV. EXPERIMENT
- Zero-shot ex-vivo deployments on unseen porcine colons cover straight, curved, and tortuous anatomies.Source: §IV · p. 4
- The evaluation compares input-cue ablations and an RGB-D-E behavior-cloning baseline.Source: §IV · p. 4
- Held-out offline ablations assess control stability and long-horizon prediction fidelity using ADE/FDE, alongside open-loop rollouts.Source: §IV · p. 4
1) Training Setting:
The section reports ColoACT training and ex-vivo validation, with zero-shot success across straight, curved, and tortuous porcine-colon segments and offline assessments of control stability and rollouts.
1) Training Setting:
- 214 teleoperated trajectories covered straight, turning, recovery, and occlusion scenarios; training used 100 epochs, batch size 64, and AdamW with a 10^-4 learning rate.Weight decay was 10^-4 and the KL-loss weight was 10.0.Source: 1) Training Setting · p. 4
2) Ex-Vivo Evaluation of Autonomous Navigation in Porcine Colon:
- Validation used excised porcine large intestine because it preserves nonlinear viscoelastic tissue characteristics and resembles human-colon anatomy more closely than silicone phantoms.The passage identifies haustra and teniae coli as relevant shared anatomical features.Source: 2) Ex-Vivo Evaluation of Autonomous… · p. 5
- 85.4% straight and 72.5% curved success: RGB-D-E ACT outperformed RGB-only ACT (35.4%, 25.0%) and RGB-D ACT (58.3%, 50.0%) on four unseen colons.The four approximately 60 cm porcine colons were excluded from training; the protocol used 48 straight and 40 curved trials.Source: 2) Ex-Vivo Evaluation of Autonomous… · p. 5, Table I · p. 5
- With RGB-D-E fixed, non-temporal BC reached 45.8% straight and 57.5% curved success, with angular jerk of about 60 a.u. versus ACT below 5 a.u.The reported comparison describes BC as producing high-frequency oscillations, while ColoACT moved smoothly under weak-texture conditions.Source: 2) Ex-Vivo Evaluation of Autonomous… · p. 5, Figure 3 · p. 5
- Across double- and triple-bend sequences, angular-velocity polarity reversed across successive bends, with the reported control switching described as continuous and oscillation-free.In the double-bend example, the output mirrors the S-shaped geometry; in the triple bend, polarity changes across all three bends.Source: 2) Ex-Vivo Evaluation of Autonomous… · p. 6, Figure 4 · p. 6
- In complex maneuvers, ColoACT achieved success rates of 70% (7/10) in 90-degree turns and 60% (6/10) in double-bend sequences, with 30% (3/10) feasible passage in triple-bend anatomies.Source: 2) Ex-Vivo Evaluation of Autonomous… · p. 6
- Temporal ensembling with k = 16 suppresses jitter compared with the k = 1 baseline; open-loop rollouts compare ACT predictions with demonstrations in normalized kinematic space.Source: Figure 5 · p. 6
3) Ablations on Temporal Ensembling and Input Cues, with Long-Horizon Open-Loop Rollout Validation:
The ablations show that temporal ensembling smooths angular-velocity commands, while adding elevation maps improves open-loop prediction fidelity. Qualitative rollouts capture expert-path curvature and provide geometric priors for closed-loop tracking.
Temporal ensembling
- With k = 16, temporal ensembling filters high-frequency oscillation in angular-velocity commands compared with raw predictions at k = 1.The validation-set comparison used N = 43; the authors note that oscillation risks motor service life and video stability.Source: 3) Ablations on Temporal Ensembling… · p. 6
Input-cue ablation
- RGB-D-E reduced ADE by 23.8% and FDE by 18.6% versus the RGB-D baseline in open-loop prediction.The authors associate the reductions with tighter alignment to expert geometric intent and a more stable prediction tail.Source: 3) Ablations on Temporal Ensembling… · p. 7
Long-horizon rollout validation
- Open-loop rollouts capture the curvature of expert paths, and their long-term predictions serve as geometric priors for receding-horizon closed-loop tracking.The rollouts use single-instance chunks because localization is absent; open-loop drift remains inherent.Source: 3) Ablations on Temporal Ensembling… · p. 7
4) Semantic and Attention Analysis of Policy Behavior:
The latent representation is organized along steering intent, while Grad-CAM visualizations show attention to navigable lumen and nearby anatomical boundaries during turning.
Latent-space semantics
- The latent space forms a continuous manifold spanning negative to positive angular velocities, organized by steering intent.The authors interpret this pattern as evidence that the CVAE encoder disentangles geometric features and supports continuous, stable controls, including in unseen scenarios.Source: 4) Semantic and Attention Analysis of… · p. 7
Spatial attention
- During a left turn, Grad-CAM highlights the distant navigable lumen and also attends to adjacent haustral fold ridges.The authors interpret the pattern as target-driven pathfinding that identifies open trajectory while tracking nearby topological boundaries.Source: 4) Semantic and Attention Analysis of… · p. 7
V. CONCLUSIONS & FUTURE WORK
ColoACT combines an RGB-D-E Action Chunking Transformer policy with a compact self-propelled BGER for autonomous navigation. Ex-vivo validation reports success rates of 85.4% in straight sections and 72.5% in curved sections; future work includes force sensing and in-vivo trials.
V. CONCLUSIONS & FUTURE WORK
- ColoACT achieved success rates of 85.4% in straight sections and 72.5% in curved sections in ex-vivo validation.The authors also report adaptability to 90-degree turns and double- and triple-bend colon segments.Source: §V · pp. 7–8
- The policy combines RGB with relative depth and pseudo-elevation cues to highlight fold ridges and other high-frequency geometry.Action chunking and temporal ensembling produce smooth, continuous motion for the BGER.Source: §V · p. 7
- Trained on expert demonstrations, the policy inherits human navigation priors and reduces reliance on modeling viscoelastic contact interactions.The system is designed for a compact self-propelled BGER.Source: §V · p. 7
- Future work plans to add force sensing for tissue-interaction safety and validate in vivo under peristalsis and varying lubrication.These trials are intended to address physiological dynamics relevant to practical clinical evaluation.Source: §V · p. 8