PaperlayerView exact revision on arXiv

Source-linked AI summary

EndoLIFT: Language-Disambiguated Latent-Conditioned Rectified Flow for Bidirectional Endoscopic Control

Chi Kit Ng, Yidong Zhang, Lui Siu Hing, Jinsong Lin, Tianchun Wu, Ho Yin Chim, Zhiqing Tang, Tao Yang, Huxin Gao, Trevor Yeung, Raymond Shing-Yan Tang, Hongliang Ren

arXiv:2608.20478v1cs.RO

TL;DR

Bidirectional endoscopy creates intent aliasing because nearly identical visual scenes can require opposite axial actions when the procedural phase changes. EndoLIFT combines explicit language conditioning with a 32-D trajectory latent and rectified-flow action generation, improving directional correctness, retraction, linguistic intent following, and closed-loop success over the matched no-VTL model. The evaluation remains bounded to phantoms and ex-vivo porcine trachea, with operator-supervised task autonomy.

  • Problem

    Nearly identical endoscopic observations can require opposite axial actions when externally specified procedural intent changes, so pixels alone cannot identify the intended mode.

  • Method

    EndoLIFT uses RGB, language, and previous-action context with a 32-D variational trajectory latent conditioning a rectified-flow Transformer for continuous action chunks.

  • Results

    Relative to EndoLIFT w/o VTL, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points, reduces wrong-direction advance by 83%, and raises closed-loop success by 30 percentage points across seen and unseen phantoms.

  • Takeaways & Limitations

    Language selects the requested axial mode, while the trajectory latent contributes to directional correctness and robust closed-loop retraction.

  • Takeaways & Limitations

    Evaluation uses phantoms with ten trials per method–condition cell and ten ex-vivo porcine-trachea trials, motivating larger synchronized closed-loop studies.

Abstract

from arXiv · show

Routine gastrointestinal endoscopy is intrinsically bidirectional: the instrument is advanced to reach target anatomy and later withdrawn or retroflexed for inspection, while an external cue may require earlier reversal. When the requested phase changes before the visual scene does, nearly identical observations can require opposite axial actions. We identify and formalize this ambiguity in bidirectional endoscopic control as intent aliasing. We propose EndoLIFT (Endoscopic Language-Instruction Flow with Trajectory Latents), a vision-language-action policy that combines explicit language-based intent conditioning with a latent-conditioned rectified-flow action expert. The policy receives RGB, a language instruction, and the previous-action state; a 32-D variational trajectory latent stochastically conditions continuous action-chunk generation. Controlled same-observation instruction swaps establish that language selects the axial mode, independently of whether the trajectory latent is present. Relative to the matched model without latent conditioning, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points and reduces wrong-direction advance by 83\%. An architecture-controlled 1-bit mode-flag reference exhibits weaker canonical-anchor switching, while EndoLIFT retains 82.8\% intent-following accuracy across 44 held-out linguistic variants. In closed-loop evaluation, EndoLIFT improves overall success by 30 percentage points over EndoLIFT w/o VTL on both the seen colon phantom and the unseen lung and stomach phantoms, and completes 10/10 ex-vivo porcine-trachea trials. These results separate language-based intent selection from the trajectory latent's contribution to directional correctness and robust retraction.

I. INTRODUCTION

Bidirectional endoscopy requires both advancement and controlled withdrawal, but visually similar scenes can demand opposite axial actions when procedural intent changes. EndoLIFT addresses this ambiguity with language-conditioned continuous control and evaluates intent switching, latent conditioning, linguistic variation, and cross-domain execution.

  • Motivation: Visually neighboring endoscopic observations can require opposite axial actions when the externally specified procedural phase changes, creating intent aliasing.Without an explicit intent variable, the observation-to-action mapping is multi-valued.
  • Motivation: Language provides a practical, auditable channel for specifying procedural intent while clinicians’ hands remain occupied with the endoscope and accessories.The system uses an external monitor–speech supervisor to select a forward or retraction instruction.
  • EndoLIFT: EndoLIFT combines RGB, canonical language, and previous-action context with a PaliGemma 2 backbone and a 32-D variational trajectory latent conditioning rectified-flow action generation.The policy generates continuous three-axis action chunks, while the external trigger supervisor remains outside the learned network.
  • Scope: The prototype provides Level 2 task autonomy: the operator initiates or verifies bounded forward/retract tasks and retains discrete intervention authority.It does not autonomously define clinical goals, generate a complete procedural strategy, or claim whole-examination autonomy.
  • Evaluation: The evaluation isolates language-based mode selection from trajectory-latent effects using fixed-observation instruction swaps, a ModeFlag reference, a no-VTL ablation, held-out linguistic variants, and phantom transfer.Colon is the seen domain, while lung and stomach phantoms are unseen domains.

II. RELATED WORK

Related work spans language-conditioned robot learning, embodied multimodal models, visual representations, medical endoscopy, and multimodal action generation. EndoLIFT differs by using language specifically to select between opposing axial modes while combining it with latent-conditioned continuous trajectory generation.

  • Generalist robot learning: Generalist robot-learning systems study language or multimodal prompting for task generalization, spatial manipulation, long-horizon execution, and cross-robot transfer.Examples include BC-Z, CLIPort, CALVIN, VIMA, PaLM-E, Gato, RT-1, RT-2, OpenVLA, Open X-Embodiment, and Octo.
  • Robotic endoscopy: The evaluated platform couples robotic feeding for longitudinal advance/retraction with two-axis distal bending, matching the policy’s three action channels.Colon is the seen phantom, while lung and stomach phantoms are held out as unseen geometries.
  • Medical robotics: Medical-robotics work includes prompt-conditioned tracking, scene-aware navigation, grounded world–action modeling, and large-scale medical-robotics datasets.These instructions generally organize tasks or visually recognized stages.
  • Action generation: Behavior Transformers and Diffusion Policy represent multimodal continuous behavior, while ACT, flow matching, rectified flow, and π0 support chunked or flow-based action generation.These approaches can represent multiple plausible trajectories but do not identify which externally requested procedural phase is active.

III. PROBLEM FORMULATION

The formulation treats forward navigation and urgent retraction as distinct modes that can share nearly identical observations. It motivates explicit language conditioning and combines it with a latent-conditioned rectified-flow policy for continuous action chunks.

  • Variables: The observation consists of an RGB frame and previous normalized action state, while the mode denotes normal navigation or urgent retraction.The target chunk has three action dimensions: longitudinal motion and two bending directions.
  • Variables: Longitudinal action uses the sign convention afwd < 0 for advance and afwd > 0 for retraction.This convention defines the axial direction used in the formulation.
  • Intent aliasing: When both modes occur near the same observation, squared-error regression combines opposing actions according to the probability of urgent retraction.Class imbalance can bias regression toward one mode, while balanced modes can produce averaging or an unhelpful hold.
  • Policy: EndoLIFT conditions an eight-block rectified-flow action expert on visual-language context, previous action, noisy action chunks, flow time, and a trajectory latent.The velocity field is integrated for 20 Euler steps to produce a normalized 32×3 action chunk.
  • Evaluation: The evaluation holds image and previous action fixed while switching mode conditions, then compares latent conditioning and closed-loop performance across seen and unseen phantoms.ModeFlag-LCRF separately tests whether a literal one-bit interface suffices when an upstream mode is known.

IV. METHOD

EndoLIFT separates mode selection from visual control: an external supervisor chooses one of two instructions, and the VLA generates geometry-dependent continuous motion conditioned on RGB, language, previous action, and a trajectory latent.

  • The external supervisor selects one of two fixed instructions, after which the same VLA produces geometry-dependent motion.
  • The policy combines a PaliGemma 2 visual-language prefix with a rectified-flow Transformer action expert and previous-action state.
  • The three continuous action channels control longitudinal motion, distal up–down bending, and distal left–right bending.
  • The 32-step action horizon is replanned after at most eight executed actions, while an instruction change invalidates the active chunk.
  • A 32-D variational trajectory latent conditions flow-time normalization and stochastic action-chunk generation.
  • At inference, latent sampling and 20 Euler steps transform an initially noisy chunk into the executed action prediction.

E. Architecture-Controlled ModeFlag Reference

ModeFlag-LCRF provides an architecture-controlled one-bit alternative to text conditioning while preserving the visual pathway, trajectory latent, and rectified-flow action expert.

  • ModeFlag-LCRF removes instruction text and uses a constant neutral prompt for both modes.
  • A learned embedding and MLP map the literal advance/retract bit to a prefix token and adaptive-normalization condition.
  • The reference retains PaliGemma 2 vision, previous-action input, a 32-D trajectory latent, and the matched width-1024 rectified-flow expert.
  • ModeFlag-LCRF is fine-tuned for two epochs on frozen-teacher action chunks from 29,672 colon frames, optimizing its mode encoder and action expert while LoRA remains fixed.
  • The base imitation dataset contains four instruction surface forms corresponding to two axial control modes.

F. External Trigger Supervisor and Instruction Switching

The supervisor converts physiological and speech events into canonical instruction switches, while the evaluation tests language conditioning, baselines, and held-out linguistic variants across the defined control setup.

  • F. External Trigger Supervisor and Instruction Switching: Physiological thresholds and a two-stage speech path generate discrete events outside the VLA, triggering forward navigation or urgent retraction.
  • F. External Trigger Supervisor and Instruction Switching: Resumption requires normalized physiology and a high-confidence KEEP FORWARD event within a finite confirmation window.
  • A. Robotic Platform and Phantom Domains: The platform uses a 480×480 tip camera and 20-Hz host and motor interfaces, with colon seen and lung and stomach unseen phantom domains.
  • B. Tasks, Instructions, and Data Composition: The base dataset contains 44,942 frames and four instruction strings spanning forward and retraction modes.
  • B. Tasks, Instructions, and Data Composition: Instruction conditioning uses 1,200 sampled frames, 40 forward and 40 retraction paraphrases, perturbations, and a 20% canonical-anchor mixture.
  • C. Baselines and Ablations: EndoLIFT w/o VTL is the matched PaliGemma 2 rectified-flow ablation, whereas EndoLIFT adds the 32-D trajectory latent.
  • D. Mode-Interface and Instruction-Generalization Evaluation: The architecture-controlled comparison freezes images, previous actions, and flow-noise seeds while measuring intent-following and canonical paired flips.
  • D. Mode-Interface and Instruction-Generalization Evaluation: Instruction generalization tests 44 held-out commands across English, Chinese, oral, noisy/ASR, and distractor-containing variants.

E. Policy-Level Evaluation

Policy-level evaluation combines frozen-input interventions, direction and retraction metrics, closed-loop phantom trials, and triggered porcine-trachea tests.

  • The episode-level split contains 166 training, nine validation, and ten test episodes with no episode overlap across partitions.
  • Navigation direction accuracy measures three-way advance/hold/retract classification, while retraction precision, recall, F1, and critical advance rate assess retraction behavior.
  • Conditioning interventions swap instructions with image and previous action fixed, replace text with ModeFlag, or shuffle images while retaining mode and state.
  • Closed-loop testing uses ten trials per method-condition cell across seen colon and unseen lung and stomach domains.
  • Table II compares canonical-text and ModeFlag interventions on 40 frozen frames with identical images, previous actions, and flow-noise seeds.
  • Success rates use exact 95% Clopper–Pearson intervals, while EndoLIFT and EndoLIFT w/o VTL are compared with two-sided Fisher exact tests.
  • Ten triggered-retraction trials in a porcine trachea measure success, trigger-to-action latencies, post-trigger direction, residual bending, and human takeover.

VI. RESULTS

EndoLIFT separates instruction-driven axial-mode selection from trajectory-latent-driven execution quality. It outperforms the matched no-VTL model on directional control while preserving instruction sensitivity across held-out language variants.

  • Literal 1-bit baseline: 16.2 percentage points and 32.5 points are the canonical-text gains in IFA and paired flip over the literal bit, respectively.The mean conditional separation is also 2.86× larger under canonical text.
  • Held-out instructions: 82.8% IFA across 44 held-out surface forms exceeds EndoLIFT w/o VTL at 78.2%.Qwen3-VL + GR00T and Qwen3-VL + MLP achieve 63.1% and 59.7%, respectively.
  • Instruction swaps: Changing only the instruction shifts EndoLIFT’s longitudinal output 3.484 raw motor units toward retraction, with 68.1% of paired predictions moving correctly.The diagnostic holds image and previous-action state fixed; the matched no-VTL model shows a larger response but stronger task results belong to EndoLIFT.
  • Visual conditioning: 13.742 to 14.148 overall bending MAE after image shuffling demonstrates measurable visual dependence of bending predictions.Forward-navigation bending MAE rises from 32.927 to 34.227.
  • Latent conditioning: 11.1 percentage points higher navigation-direction accuracy and 83% lower wrong-direction advance distinguish EndoLIFT from EndoLIFT w/o VTL.The models share the backbone, state input, action horizon, and flow objective, differing in trajectory-latent conditioning.

E. Action Chunks Preserve Three-Axis Temporal Structure

EndoLIFT preserves temporal structure across axial and bending channels during both navigation and urgent retraction. Closed-loop evaluations show that its advantage is concentrated in reliable reversal rather than forward navigation.

  • Action chunks: EndoLIFT follows forward-navigation axial plateaus and returns toward zero during inactive intervals.The no-VTL model underestimates early and late axial segments and prolongs bending beyond target intervals.
  • Action chunks: During urgent retraction, EndoLIFT aligns with three positive axial events and reproduces dominant negative-bending timing.The no-VTL and generic baselines suppress or underestimate retraction events, or oscillate on bending axes.
  • Transition structure: Under urgent retraction, monitor and speech triggers enter the retraction subgraph, connecting Hold and bending states to Retract.Forward-navigation transitions instead converge on Advance, with bending returning to forward motion.
  • Seen-domain closed loop: 30 percentage points higher overall success on the seen colon phantom comes from a 60-percentage-point increase in urgent-retraction success, while navigation remains unchanged.EndoLIFT achieves 10/10 retraction versus 4/10 for EndoLIFT w/o VTL.
  • Unseen-domain transfer: 30 percentage points higher overall success on unseen lung and stomach phantoms accompanies a 60-percentage-point gain in retraction success.Pooled unseen retraction is 20/20 for EndoLIFT versus 8/20 for the matched ablation.

I. Demonstrations Resolve Intent Aliasing and Execute Triggered Reversal

When the instruction changes under nearly unchanged visual conditions, EndoLIFT reverses axial motion while continuing camera-conditioned bending. This behavior transfers across phantoms and succeeds in all ten ex-vivo trials.

  • Triggered reversal: After the instruction changes, the robot retracts although the endoscopic view is nearly unchanged.Camera feedback continues to condition bending during reversed axial motion.
  • Cross-domain transitions: Across all three phantom domains, camera-conditioned bending and hold states connect to both axial Advance and Retract states.Monitor and speech routes provide the external mode interface, and bending-to-axial transitions occur in both modes.
  • Ex-vivo validation: 10/10 ex-vivo porcine-trachea trials succeed, with the first retract command following the instruction switch in 54 ± 61 ms.All 1,127 post-trigger axial-dominant steps are Retract, and no trial requires human takeover.
  • Failure case: EndoLIFT w/o VTL retains forward navigation but has a 60-percentage-point retraction deficit on seen and unseen evaluations.Its urgent-retraction chunk underestimates axial peaks and introduces positive pre-event bias.

VIII. DISCUSSION

EndoLIFT separates language-based axial-mode selection from trajectory execution, combining a language-conditioned policy with a variational trajectory latent for bidirectional control. It improves directional and closed-loop performance across seen, unseen, and ex-vivo settings, while the evaluation remains limited in scale.

  • Language and VTL: Language changes the requested axial mode under fixed visual state, while VTL primarily improves trajectory execution rather than mode selection.The fixed-observation intervention and matched no-VTL comparison identify complementary roles for language and VTL.
  • Limitations: The evaluation uses ten trials per phantom method-condition cell and ten ex-vivo porcine-trachea trials, motivating larger repeated closed-loop studies with synchronized measurements.The proposed future measurements include video, action, trigger, and force signals.
  • Architecture: EndoLIFT combines a PaliGemma 2-based vision-language-action policy with a 32-D variational trajectory latent and rectified-flow Transformer for receding-horizon endoscope control.A separate external trigger supervisor switches the instruction outside the learned network.
  • Language beyond one bit: EndoLIFT retains 82.8% intent-following accuracy across 44 held-out linguistic variants, while its text-conditioned policy produces a larger mode-separation response than ModeFlag-LCRF.The comparison tests language-based switching against an architecture-controlled one-bit mode interface.
  • Directional performance: 11.1 percentage points higher navigation-direction accuracy and 83% lower wrong-direction advance distinguish EndoLIFT from EndoLIFT w/o VTL.These results support a contribution from latent conditioning to directional correctness and retraction behavior.
  • Closed-loop transfer: 30 percentage points higher closed-loop success over EndoLIFT w/o VTL occurs in both the seen colon phantom and unseen lung and stomach phantoms.The unseen phantoms serve as out-of-domain luminal transfer tests beyond training-domain colon geometry.
Loading 2608.20478v1…