PaperlayerView exact revision on arXiv

Source-linked AI summary

EndoLIFT: Language-Disambiguated Latent-Conditioned Rectified Flow for Bidirectional Endoscopic Control

Chi Kit Ng, Yidong Zhang, Lui Siu Hing, Jinsong Lin, Tianchun Wu, Ho Yin Chim, Zhiqing Tang, Tao Yang, Huxin Gao, Trevor Yeung, Raymond Shing-Yan Tang, Hongliang Ren

arXiv:2608.20478v1cs.RO

TL;DR

Bidirectional endoscopy creates intent aliasing because nearly identical visual scenes can require opposite axial actions when the procedural phase changes. EndoLIFT combines explicit language conditioning with a 32-D trajectory latent and rectified-flow action generation, improving directional correctness, retraction, linguistic intent following, and closed-loop success over the matched no-VTL model. The evaluation remains bounded to phantoms and ex-vivo porcine trachea, with operator-supervised task autonomy.

  • Problem

    Nearly identical endoscopic observations can require opposite axial actions when externally specified procedural intent changes, so pixels alone cannot identify the intended mode.

    Source: §I · p. 1

  • Method

    EndoLIFT uses RGB, language, and previous-action context with a 32-D variational trajectory latent conditioning a rectified-flow Transformer for continuous action chunks.

    Source: §I · pp. 1–2

  • Results

    Relative to EndoLIFT w/o VTL, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points, reduces wrong-direction advance by 83%, and raises closed-loop success by 30 percentage points across seen and unseen phantoms.

    Source: §IX · p. 12

  • Takeaways & Limitations

    Language selects the requested axial mode, while the trajectory latent contributes to directional correctness and robust closed-loop retraction.

    Source: §VIII · pp. 10–11

    Evaluation uses phantoms with ten trials per method–condition cell and ten ex-vivo porcine-trachea trials, motivating larger synchronized closed-loop studies.

    Source: §VIII · p. 11

Abstract

from arXiv · show

Routine gastrointestinal endoscopy is intrinsically bidirectional: the instrument is advanced to reach target anatomy and later withdrawn or retroflexed for inspection, while an external cue may require earlier reversal. When the requested phase changes before the visual scene does, nearly identical observations can require opposite axial actions. We identify and formalize this ambiguity in bidirectional endoscopic control as intent aliasing. We propose EndoLIFT (Endoscopic Language-Instruction Flow with Trajectory Latents), a vision-language-action policy that combines explicit language-based intent conditioning with a latent-conditioned rectified-flow action expert. The policy receives RGB, a language instruction, and the previous-action state; a 32-D variational trajectory latent stochastically conditions continuous action-chunk generation. Controlled same-observation instruction swaps establish that language selects the axial mode, independently of whether the trajectory latent is present. Relative to the matched model without latent conditioning, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points and reduces wrong-direction advance by 83\%. An architecture-controlled 1-bit mode-flag reference exhibits weaker canonical-anchor switching, while EndoLIFT retains 82.8\% intent-following accuracy across 44 held-out linguistic variants. In closed-loop evaluation, EndoLIFT improves overall success by 30 percentage points over EndoLIFT w/o VTL on both the seen colon phantom and the unseen lung and stomach phantoms, and completes 10/10 ex-vivo porcine-trachea trials. These results separate language-based intent selection from the trajectory latent's contribution to directional correctness and robust retraction.

I. INTRODUCTION

Bidirectional endoscopy requires both advancement and controlled withdrawal, but visually similar scenes can demand opposite axial actions when procedural intent changes. EndoLIFT addresses this ambiguity with language-conditioned continuous control and evaluates intent switching, latent conditioning, linguistic variation, and cross-domain execution.

Motivation

  • Visually neighboring endoscopic observations can require opposite axial actions when the externally specified procedural phase changes, creating intent aliasing.Without an explicit intent variable, the observation-to-action mapping is multi-valued.Source: §I · p. 1
  • Language provides a practical, auditable channel for specifying procedural intent while clinicians’ hands remain occupied with the endoscope and accessories.The system uses an external monitor–speech supervisor to select a forward or retraction instruction.Source: §I · p. 1, Figure 1 · p. 2

EndoLIFT

  • EndoLIFT combines RGB, canonical language, and previous-action context with a PaliGemma 2 backbone and a 32-D variational trajectory latent conditioning rectified-flow action generation.The policy generates continuous three-axis action chunks, while the external trigger supervisor remains outside the learned network.Source: §I · pp. 1–2

Scope

  • The prototype provides Level 2 task autonomy: the operator initiates or verifies bounded forward/retract tasks and retains discrete intervention authority.It does not autonomously define clinical goals, generate a complete procedural strategy, or claim whole-examination autonomy.Source: §I · p. 2

Evaluation

  • The evaluation isolates language-based mode selection from trajectory-latent effects using fixed-observation instruction swaps, a ModeFlag reference, a no-VTL ablation, held-out linguistic variants, and phantom transfer.Colon is the seen domain, while lung and stomach phantoms are unseen domains.Source: §I · p. 2

II. RELATED WORK

Related work spans language-conditioned robot learning, embodied multimodal models, visual representations, medical endoscopy, and multimodal action generation. EndoLIFT differs by using language specifically to select between opposing axial modes while combining it with latent-conditioned continuous trajectory generation.

Generalist robot learning

  • Generalist robot-learning systems study language or multimodal prompting for task generalization, spatial manipulation, long-horizon execution, and cross-robot transfer.Examples include BC-Z, CLIPort, CALVIN, VIMA, PaLM-E, Gato, RT-1, RT-2, OpenVLA, Open X-Embodiment, and Octo.Source: B. Language-Conditioned VLA for… · pp. 2–3

Robotic endoscopy

  • The evaluated platform couples robotic feeding for longitudinal advance/retraction with two-axis distal bending, matching the policy’s three action channels.Colon is the seen phantom, while lung and stomach phantoms are held out as unseen geometries.Source: Figure 2 · p. 3

Medical robotics

  • Medical-robotics work includes prompt-conditioned tracking, scene-aware navigation, grounded world–action modeling, and large-scale medical-robotics datasets.These instructions generally organize tasks or visually recognized stages.Source: B. Language-Conditioned VLA for… · p. 3

Action generation

  • Behavior Transformers and Diffusion Policy represent multimodal continuous behavior, while ACT, flow matching, rectified flow, and π0 support chunked or flow-based action generation.These approaches can represent multiple plausible trajectories but do not identify which externally requested procedural phase is active.Source: C. Generative Action Modeling and… · p. 3

III. PROBLEM FORMULATION

The formulation treats forward navigation and urgent retraction as distinct modes that can share nearly identical observations. It motivates explicit language conditioning and combines it with a latent-conditioned rectified-flow policy for continuous action chunks.

Variables

  • The observation consists of an RGB frame and previous normalized action state, while the mode denotes normal navigation or urgent retraction.The target chunk has three action dimensions: longitudinal motion and two bending directions.Source: §III · p. 3
  • Longitudinal action uses the sign convention afwd < 0 for advance and afwd > 0 for retraction.This convention defines the axial direction used in the formulation.Source: §III · p. 3

Intent aliasing

  • When both modes occur near the same observation, squared-error regression combines opposing actions according to the probability of urgent retraction.Class imbalance can bias regression toward one mode, while balanced modes can produce averaging or an unhelpful hold.Source: §III · p. 3

Policy

  • EndoLIFT conditions an eight-block rectified-flow action expert on visual-language context, previous action, noisy action chunks, flow time, and a trajectory latent.The velocity field is integrated for 20 Euler steps to produce a normalized 32×3 action chunk.Source: Figure 3 · p. 4

Evaluation

  • The evaluation holds image and previous action fixed while switching mode conditions, then compares latent conditioning and closed-loop performance across seen and unseen phantoms.ModeFlag-LCRF separately tests whether a literal one-bit interface suffices when an upstream mode is known.Source: §III · p. 4

IV. METHOD

EndoLIFT separates mode selection from visual control: an external supervisor chooses one of two instructions, and the VLA generates geometry-dependent continuous motion conditioned on RGB, language, previous action, and a trajectory latent.

E. Architecture-Controlled ModeFlag Reference

ModeFlag-LCRF provides an architecture-controlled one-bit alternative to text conditioning while preserving the visual pathway, trajectory latent, and rectified-flow action expert.

F. External Trigger Supervisor and Instruction Switching

The supervisor converts physiological and speech events into canonical instruction switches, while the evaluation tests language conditioning, baselines, and held-out linguistic variants across the defined control setup.

F. External Trigger Supervisor and Instruction Switching

A. Robotic Platform and Phantom Domains

B. Tasks, Instructions, and Data Composition

C. Baselines and Ablations

D. Mode-Interface and Instruction-Generalization Evaluation

  • The architecture-controlled comparison freezes images, previous actions, and flow-noise seeds while measuring intent-following and canonical paired flips.Source: D. Mode-Interface and… · p. 6
  • Instruction generalization tests 44 held-out commands across English, Chinese, oral, noisy/ASR, and distractor-containing variants.Source: D. Mode-Interface and… · p. 6

E. Policy-Level Evaluation

Policy-level evaluation combines frozen-input interventions, direction and retraction metrics, closed-loop phantom trials, and triggered porcine-trachea tests.

  • The episode-level split contains 166 training, nine validation, and ten test episodes with no episode overlap across partitions.Source: E. Policy-Level Evaluation · p. 6
  • Navigation direction accuracy measures three-way advance/hold/retract classification, while retraction precision, recall, F1, and critical advance rate assess retraction behavior.Source: E. Policy-Level Evaluation · p. 6
  • Conditioning interventions swap instructions with image and previous action fixed, replace text with ModeFlag, or shuffle images while retaining mode and state.Source: E. Policy-Level Evaluation · p. 6
  • Closed-loop testing uses ten trials per method-condition cell across seen colon and unseen lung and stomach domains.Source: F. Closed-Loop Phantom Evaluation · p. 6
  • Table II compares canonical-text and ModeFlag interventions on 40 frozen frames with identical images, previous actions, and flow-noise seeds.Source: Table II · p. 7
  • Success rates use exact 95% Clopper–Pearson intervals, while EndoLIFT and EndoLIFT w/o VTL are compared with two-sided Fisher exact tests.Source: F. Closed-Loop Phantom Evaluation · p. 7
  • Ten triggered-retraction trials in a porcine trachea measure success, trigger-to-action latencies, post-trigger direction, residual bending, and human takeover.Source: G. Ex-Vivo Porcine-Trachea Evaluation · p. 7

VI. RESULTS

EndoLIFT separates instruction-driven axial-mode selection from trajectory-latent-driven execution quality. It outperforms the matched no-VTL model on directional control while preserving instruction sensitivity across held-out language variants.

Literal 1-bit baseline

  • 16.2 percentage points and 32.5 points are the canonical-text gains in IFA and paired flip over the literal bit, respectively.The mean conditional separation is also 2.86× larger under canonical text.Source: A. Literal 1-bit Baseline and Held-Out… · p. 7

Held-out instructions

Instruction swaps

  • Changing only the instruction shifts EndoLIFT’s longitudinal output 3.484 raw motor units toward retraction, with 68.1% of paired predictions moving correctly.The diagnostic holds image and previous-action state fixed; the matched no-VTL model shows a larger response but stronger task results belong to EndoLIFT.Source: B. Language Selects the Axial Motion… · p. 7, Table IV · p. 8

Visual conditioning

  • 13.742 to 14.148 overall bending MAE after image shuffling demonstrates measurable visual dependence of bending predictions.Forward-navigation bending MAE rises from 32.927 to 34.227.Source: C. Visual Observations Condition… · p. 7

Latent conditioning

  • 11.1 percentage points higher navigation-direction accuracy and 83% lower wrong-direction advance distinguish EndoLIFT from EndoLIFT w/o VTL.The models share the backbone, state input, action horizon, and flow objective, differing in trajectory-latent conditioning.Source: D. Latent Conditioning Improves… · pp. 7–8

E. Action Chunks Preserve Three-Axis Temporal Structure

EndoLIFT preserves temporal structure across axial and bending channels during both navigation and urgent retraction. Closed-loop evaluations show that its advantage is concentrated in reliable reversal rather than forward navigation.

Action chunks

  • EndoLIFT follows forward-navigation axial plateaus and returns toward zero during inactive intervals.The no-VTL model underestimates early and late axial segments and prolongs bending beyond target intervals.Source: E. Action Chunks Preserve Three-Axis… · p. 8
  • During urgent retraction, EndoLIFT aligns with three positive axial events and reproduces dominant negative-bending timing.The no-VTL and generic baselines suppress or underestimate retraction events, or oscillate on bending axes.Source: E. Action Chunks Preserve Three-Axis… · p. 8

Transition structure

Seen-domain closed loop

  • 30 percentage points higher overall success on the seen colon phantom comes from a 60-percentage-point increase in urgent-retraction success, while navigation remains unchanged.EndoLIFT achieves 10/10 retraction versus 4/10 for EndoLIFT w/o VTL.Source: G. Latent Conditioning Improves… · p. 8

Unseen-domain transfer

  • 30 percentage points higher overall success on unseen lung and stomach phantoms accompanies a 60-percentage-point gain in retraction success.Pooled unseen retraction is 20/20 for EndoLIFT versus 8/20 for the matched ablation.Source: H. The Bidirectional Policy Transfers… · p. 8

I. Demonstrations Resolve Intent Aliasing and Execute Triggered Reversal

When the instruction changes under nearly unchanged visual conditions, EndoLIFT reverses axial motion while continuing camera-conditioned bending. This behavior transfers across phantoms and succeeds in all ten ex-vivo trials.

Triggered reversal

  • After the instruction changes, the robot retracts although the endoscopic view is nearly unchanged.Camera feedback continues to condition bending during reversed axial motion.Source: §I · p. 9

Cross-domain transitions

Ex-vivo validation

Failure case

  • EndoLIFT w/o VTL retains forward navigation but has a 60-percentage-point retraction deficit on seen and unseen evaluations.Its urgent-retraction chunk underestimates axial peaks and introduces positive pre-event bias.Source: §VII · p. 10

VIII. DISCUSSION

EndoLIFT separates language-based axial-mode selection from trajectory execution, combining a language-conditioned policy with a variational trajectory latent for bidirectional control. It improves directional and closed-loop performance across seen, unseen, and ex-vivo settings, while the evaluation remains limited in scale.

Language and VTL

  • Language changes the requested axial mode under fixed visual state, while VTL primarily improves trajectory execution rather than mode selection.The fixed-observation intervention and matched no-VTL comparison identify complementary roles for language and VTL.Source: §VIII · p. 11

Limitations

  • The evaluation uses ten trials per phantom method-condition cell and ten ex-vivo porcine-trachea trials, motivating larger repeated closed-loop studies with synchronized measurements.The proposed future measurements include video, action, trigger, and force signals.Source: §VIII · p. 11

Architecture

  • EndoLIFT combines a PaliGemma 2-based vision-language-action policy with a 32-D variational trajectory latent and rectified-flow Transformer for receding-horizon endoscope control.A separate external trigger supervisor switches the instruction outside the learned network.Source: §IX · pp. 11–12

Language beyond one bit

  • EndoLIFT retains 82.8% intent-following accuracy across 44 held-out linguistic variants, while its text-conditioned policy produces a larger mode-separation response than ModeFlag-LCRF.The comparison tests language-based switching against an architecture-controlled one-bit mode interface.Source: §VIII · p. 10, §IX · p. 12

Directional performance

  • 11.1 percentage points higher navigation-direction accuracy and 83% lower wrong-direction advance distinguish EndoLIFT from EndoLIFT w/o VTL.These results support a contribution from latent conditioning to directional correctness and retraction behavior.Source: §VIII · p. 11, §IX · p. 12

Closed-loop transfer

  • 30 percentage points higher closed-loop success over EndoLIFT w/o VTL occurs in both the seen colon phantom and unseen lung and stomach phantoms.The unseen phantoms serve as out-of-domain luminal transfer tests beyond training-domain colon geometry.Source: Table VIII · p. 11, §IX · p. 12

Related papers

Loading 2608.20478v1…