PaperlayerView exact revision on arXiv

Source-linked AI summary

LCLA: Language-Conditioned Latent Alignment for Vision-Language Navigation

Nitesh Subedi, Adam Haroon, Samuel Tetteh, Prajwal Koirala, Cody Fleming, Soumik Sarkar

arXiv:2602.07629v2cs.RO

TL;DR

Navigation must infer compact control-relevant structure from high-dimensional sensory inputs, compounded in vision-language settings by visual and linguistic uncertainty. LCLA freezes a privileged expert and trains an adapter to map observations into its latent space; it reports strong performance across in- and out-of-distribution environments. The paper frames this stable interface as supporting expert-controller reuse across modalities and environments.

  • Problem

    Navigation requires recovering compact control-relevant representations from high-dimensional sensory inputs, while vision-language navigation adds visual uncertainty and linguistic ambiguity.

    Source: §I · p. 1

  • Method

    LCLA separates a fixed privileged expert from a trainable adapter that maps image-language inputs into the expert’s internal representation.

    Source: A. Overview: Action–Perception… · p. 3

  • Results

    LCLA achieves higher success rates, more efficient trajectories, and lower collision rates than the ablated alternatives across in-distribution and out-of-distribution environments, without privileged information at test time.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The paper argues that a stable latent interface supports reusing expert controllers across sensing modalities, perception backbones, and environments.

    Source: §VII · p. 9

    The frozen controller inherits the expert’s capabilities and does not address tasks requiring exploration, memory, or long-horizon planning beyond its scope.

    Source: §VI · p. 8

Abstract

from arXiv · show

We propose LCLA (Language-Conditioned Latent Alignment), a framework for vision-language navigation that learns modular perception-action interfaces by aligning sensory observations to a latent representation of an expert policy. The expert is first trained with privileged state information, inducing a latent space sufficient for control, after which its latent interface and action head are frozen. A lightweight adapter is then trained to map raw visual-language observations, via a frozen vision-language model, into the expert's latent space, reducing the problem of visuomotor learning to supervised latent alignment rather than end-to-end policy optimization. This decoupling enforces a stable contract between perception and control, enabling expert behavior to be reused across sensing modalities and environmental variations. We instantiate LCLA and evaluate it on a vision-language indoor navigation task, where aligned latent spaces yield strong in-distribution performance and robust zero-shot generalization to unseen environments, lighting conditions, and viewpoints while remaining lightweight at inference time.

I. INTRODUCTION

LCLA treats vision-language navigation as a representation-learning problem: it aligns observations to a fixed expert latent instead of learning control end to end. The introduction motivates this design through perception-control mismatch and prior approaches’ limits, and reports zero-shot generalization to unseen scenes and objects.

I. INTRODUCTION

  • Navigation requires inferring compact geometric control variables from high-dimensional RGB and language inputs, with viewpoint, lighting, clutter, and linguistic ambiguity adding difficulty.Source: §I · pp. 1–2

A. Visual Navigation from Raw Observations

  • End-to-end pixel-to-action policies often generalize poorly and require many samples, while learned representations can exploit environment-specific correlations instead of task-relevant structure.Source: §I · p. 2

I. INTRODUCTION

  • LCLA freezes a privileged expert’s latent interface and action head, then trains a vision-language adapter to align observations with the expert latent rather than imitate actions.The fixed, task-centric latent is intended to support reuse across sensing modalities and environments without retraining control.Source: §I · p. 2
  • Despite training in one environment with limited object diversity, LCLAA generalizes zero-shot to unseen scenes and objects.Source: §I · p. 2
  • Ablations show that direct action prediction and removing explicit language conditioning each underperform LCLA by a substantial margin.Source: §I · p. 2

B. Privileged Information and Asymmetric Learning

C. Representation Learning for Control

  • Representation-learning methods improve robustness or learn compact latents, but typically couple representations to the policy, observation modality, and training environment.Source: C. Representation Learning for Control · p. 2

D. Vision–Language Navigation

  • Vision-language navigation methods commonly use VLM features as generic inputs to end-to-end policies, leaving alignment with task-optimal control representations implicit.Source: D. Vision–Language Navigation · p. 2

E. Latent Interfaces Between Perception and Control

LCLA uses a privileged expert’s predefined latent as a modular interface between perception and control. The expert learns control from compact geometric states and stays fixed while an adapter maps visual-language inputs into its control-sufficient representation.

E. Latent Interfaces Between Perception and Control

  • Unlike jointly learned latent interfaces, LCLA aligns vision-language observations to a predefined latent induced by a privileged expert, enabling modular reuse of control.Source: E. Latent Interfaces Between… · p. 3

A. Overview: Action–Perception Decoupling

  • The system separates action selection by a frozen privileged expert from a trainable adapter that translates images and language into the expert’s internal representation.The adapter is the only trainable component exposed to images and language, isolating generalization to that adapter.Source: A. Overview: Action–Perception… · p. 3

B. Stage 1: Privileged Expert Policy

  • The expert learns navigation from compact geometric state variables invariant to visual appearance, lighting, and texture, and its latent is treated as a stable control interface rather than an interpretable code.The expert is trained with reinforcement learning; the paper notes that not all intermediate representations are equally suitable for this role.Source: B. Stage 1: Privileged Expert Policy · p. 3

C. Stage 2: Visual–Linguistic Adapter

Stage 2 trains a visual-language adapter to predict the frozen expert’s latent representation from frozen VLM features, without reinforcement learning or changes to the expert. LCLAA implements this interface with language-conditioned visual selection and query-based refinement; deployment then uses its predicted latent with the frozen action head.

C. Stage 2: Visual–Linguistic Adapter

  • Stage 2 trains the adapter on expert-rollout observations and corresponding expert latents using supervised alignment; no reinforcement learning is performed, and the expert remains unchanged.The VLM’s visual and text encoders are frozen, leaving only the adapter to learn the mapping.Source: C. Stage 2: Visual–Linguistic Adapter · p. 3

D. Inference

  • At deployment, the adapter predicts an expert-compatible latent from frozen visual and language embeddings, which is passed to the frozen action head without privileged state.LCLAA contextualizes patches, selects visual context using language-conditioned scores, refines queries with cross-attention, and fuses them with a text residual.Source: D. Inference · p. 3, Figure 2 · p. 4, A. LCLAA: Language-Conditioned Latent… · p. 5

E. Key Property

  • The adapter is the only learned bridge between perception and action, separating a pixel-blind expert from a VLM not optimized for control.This bottleneck supports modularity and analysis of how perceptual errors propagate to actions.Source: E. Key Property · p. 4

IV. CONTROLLED VALIDATION STUDY

  • The controlled validation study uses LCLAA as one architecture-agnostic realization of latent alignment and aims to test latent recovery from vision-language inputs, not broad simulator-level generalization.Its language evaluation uses controlled, templated relational instructions; free-form naturalistic instructions are left for future work.Source: C. Stage 2: Visual–Linguistic Adapter · p. 3, §IV · p. 4

A. LCLAA: Language-Conditioned Latent Alignment Adapter

  • LCLAA imposes structural biases for latent alignment: language-conditioned selection suppresses spurious correlations, a bottleneck limits perception to task-relevant evidence, and cross-attention aggregates across scene complexity.The paper presents LCLAA as a concrete validation architecture, not a required architecture for the broader LCLA framework.Source: A. LCLAA: Language-Conditioned Latent… · pp. 4–5

B. Experimental Setup

The study evaluates object-goal navigation in a training environment and a held-out environment, with zero-shot evaluation in the latter. It compares LCLA with privileged-policy, behavior-cloning, and pooled-embedding baselines, using success, path efficiency, and collision metrics.

B. Experimental Setup

  • Episodes begin at random indoor poses with an instruction specifying a target category; success requires reaching a target instance within a fixed radius and time limit while facing it to a threshold.Training uses eight prompts per goal, including relational and color-attribute variations.Source: B. Experimental Setup · p. 5

B. Simulators and Environments

  • Room A is the training environment, while Room B is held out and evaluated zero-shot across changed layouts, appearance, lighting, and object categories.Room A contains 18 navigable target objects; Room B contains 12, including novel instances and previously unseen categories.Source: B. Experimental Setup · p. 5

B. Experimental Setup

  • The privileged PPO policy uses full state and serves as the in-distribution upper bound; LCBC instead predicts actions, while PELA maps pooled vision-language embeddings to expert latents.LCBC isolates the prediction-target change, and PELA isolates explicit language conditioning in the adapter.Source: B. Experimental Setup · p. 6
  • Evaluation reports Success Rate, Success weighted by Path Length, and collision rate, averaging over random seeds and initializations with 30 episodes per goal.SR measures goal reach, while SPL measures efficiency relative to the shortest path.Source: B. Experimental Setup · p. 6
  • Training uses 52k expert rollout steps and four appearance-augmented views per observation, yielding about 260k visual-language samples for ten supervised epochs.Reported results use Siglip2 base unless stated otherwise.Source: B. Experimental Setup · pp. 6–7

V. RESULTS

LCLA outperforms deployable baselines in both Room A and unseen Room B, while results also show that performance depends on latent dimensionality and the vision–language backbone.

Overall comparison

  • Across both environments and all metrics, LCLA outperforms LCBC, PELA, and LeLaN; privileged PPO is near-perfect in Room A.PPO relies on full state information, whereas the other agents are deployable.Source: §V · p. 7

In-distribution performance

  • In Room A, LCLA achieves 90.4% SR and 0.90 SPL, outperforming LCBC (60.1% SR, 0.59 SPL) and PELA (62.5% SR, 0.62 SPL).The privileged PPO policy remains superior with full-state access; LCLA narrows the gap and has lower collision rates than all baselines.Source: §V · p. 7

Out-of-distribution generalization

  • In Room B, LCLA reaches 80.5% SR and 0.80 SPL, exceeding LCBC (49.3% SR, 0.49 SPL) and PELA (57.6% SR, 0.58 SPL).It also reduces collision rates by more than 2× relative to all baselines.Source: §V · p. 7

Latent interface dimensionality

  • Performance peaks at dz = 256; both smaller and larger expert latent dimensions substantially degrade LCLA performance.Each adapter predicts the corresponding PPO hidden representation and reuses its matching frozen action head.Source: §V · p. 7

Vision–language backbone and robustness

  • Robustness to illumination and camera-pose shifts varies by frozen backbone: CLIP is sensitive, SigLIP partly robust, and SigLIP2 stable across tested conditions.The adapters and control policy are held fixed across these comparisons.Source: §V · p. 7

VI. DISCUSSION

The discussion frames LCLA as a fixed-interface approach to perception and control, not a full-stack navigation system. Its scope is bounded by the expert’s capabilities and by the tested language and scene conditions.

Task competence and deployment

  • The privileged PPO expert performs near-optimally with full state, but cannot be deployed in realistic settings that require raw sensory inputs.LCLA targets preserving expert control structure while operating from raw sensory inputs.Source: §VI · p. 7

Scope and comparison

  • LCLA is not a full-stack navigation system or benchmark, and is not directly comparable to map-centric methods that combine perception, mapping, memory, and planning.Its narrower question is whether perception can be aligned to a frozen task-centric latent interface without retraining control or constructing maps.Source: §VI · p. 7

Ablations

  • LCBC’s direct action prediction is brittle under distribution shift, while PELA’s latent alignment without language yields moderate improvement but degrades relative to LCLA.The ablations support the conclusion that neither end-to-end imitation nor latent alignment alone is sufficient.Source: §VI · p. 7

Robustness and language scope

  • LCLA preserves expert decision logic through a fixed perception–control interface, while robustness to lighting and viewpoint perturbations suggests reduced reliance on superficial visual statistics.The language evaluation uses controlled, templated instructions; more naturalistic language remains future work.Source: §VI · p. 7

Latent interface

  • The latent-dimension comparison identifies dz = 256 as critical for stable alignment; both smaller and larger dimensions cause severe out-of-distribution performance degradation.The caption describes all rows as the same LCLAA agent with different output latent dimensions.Source: Table III · p. 8

Limitations and implications

  • Stronger visual representations improve alignment without changing the controller, while the frozen policy remains limited to its expert’s capabilities and cannot handle tasks beyond its scope.The stated limitations include exploration, memory, and long-horizon planning; stability analyses are empirical, and behavior under large alignment errors is uncharacterized.Source: §VI · p. 8

VII. CONCLUSION

LCLA reuses a privileged expert by learning to recover its task-centric latent representation from vision and language. The paper reports stronger navigation across distribution shifts and presents reuse of a stable interface as its methodological claim.

Method

  • LCLA decouples perception from control by solving navigation in a compact latent space defined by a privileged expert, then recovering it from vision and language.The conclusion characterizes this as representation alignment rather than end-to-end control learning.Source: §VII · p. 8

Robustness

  • Table IV reports that stronger vision–language representations, including SigLIP and SigLIP2, improve robustness to viewpoint and lighting shifts, though all tested backbones degrade under extreme conditions.The table describes Room B perturbations.Source: Table IV · p. 8

Results

  • LCLA outperforms direct action imitation and latent alignment without language conditioning, achieving higher success, more efficient trajectories, and lower collision rates across both environments.These results do not require privileged information at test time.Source: §VII · p. 8

Implication

  • The methodological claim is that expert controllers can be reused across sensing modalities, perception backbones, and environments through a stable latent interface.The paper frames this as shifting attention from learning policies from scratch to recovering task-relevant representations.Source: §VII · pp. 8–9

Future work

  • Future work includes extending alignment to experts with memory and planning, richer naturalistic language grounding, real-world robots, and stronger theoretical characterizations.Source: §VII · p. 9

APPENDIX

The appendix describes the privileged PPO teacher’s simulation configuration, goal and reset strategy, and observation space, including auxiliary RGB and depth logging.

APPENDIX

  • The appendix documents the Isaac Lab PPO teacher configuration for Jetbot navigation, covering environment design, observations and actions, randomization, rewards, and termination conditions.Implementation-specific function and file names are omitted.Source: APPENDIX · p. 12

A. Environment and Simulation Settings

  • Training uses 16 parallel environments, 50-unit spacing, control decimation 2, a 1/60 s simulation step, and episodes capped at 15 s.The effective control step is 2/60 s; the robot uses direct differential-drive wheel-velocity targets with action scale 40.0 rad/s, velocity limits 20.0 rad/s, zero stiffness, and damping 10.0.Source: A. Environment and Simulation Settings · p. 12

B. Goal Set and Reset Strategy

  • Goals are sampled uniformly from semantic categories and candidate coordinates; the robot resets at a shared start position facing the goal with yaw noise of ±0.5 radians.A safe goal is shifted toward room center and clamped within fixed room bounds with a margin.Source: B. Goal Set and Reset Strategy · p. 12

C. Observation Space

  • The 159-dimensional privileged observation combines 9 navigation-state dimensions with 150 dimensions for 50 obstacle clusters; RGB and depth are logged but not PPO observations.Goal coordinates use the robot body frame, bearing uses sin/cos, and obstacle features are normalized by 5.0 m; obstacle centroids come from occupancy-map clustering.Source: C. Observation Space · p. 12

D. Action Space and Control

The control setup uses continuous wheel-velocity actions and reset-time domain randomization. Its shaped reward combines progress, goal alignment, obstacle and boundary penalties, and success.

D. Action Space and Control

  • The two-dimensional continuous action targets are scaled by 40.0 into wheel velocities in rad/s and applied directly to the wheel joints.Source: D. Action Space and Control · p. 12

E. Domain Randomization

  • Reset-time material randomization samples diffuse colors from [0.1, 0.9], may vary roughness and metallic, and excludes the selected goal object.Lighting randomization is supported optionally; the provided configuration uses an HDR dome light.Source: E. Domain Randomization · p. 12

F. Reward Function

  • The shaped reward combines progress and distance-potential terms, heading alignment, forward motion when aligned, obstacle and boundary penalties, and a success bonus.Source: F. Reward Function · p. 13

G. Termination and Truncation

Episodes end on success or hard collision, while timeout truncates an episode only if neither has occurred. The setup also records contact forces and evaluation-time RGB observations.

G. Termination and Truncation

H. Sensors and Auxiliary Signals

I. Architectural Details of LCLAA

LCLAA contextualizes frozen VLM patch features, selects visual regions using language, and compresses them into an instruction-conditioned bottleneck. Query refinement and gated fusion then project the visual-language representation into the expert latent space.

I. Architectural Details of LCLAA

  • Multi-head self-attention contextualizes projected visual patches, integrating local features with global scene information before instruction conditioning.Source: §I · p. 13
  • Language-conditioned patch scores are temperature-softmax normalized, with optional Gumbel noise during training to encourage stochastic exploration of patch relevance.Source: §I · p. 13
  • A convex aggregation of contextualized patches creates a single instruction-conditioned bottleneck token, restricting downstream control to a compact scene summary.Source: §I · p. 13
  • Text-derived and learnable queries are refined through cross-attention, then fused with the text embedding by a learned gate and projected into the expert latent space.Source: §I · p. 14

J. Training Objectives

LCLAA training combines supervised latent regression, contrastive alignment, and action consistency, while the reported robustness analyses test action-head stability under latent perturbations. The results are empirical rather than a formal robustness or safety guarantee.

J. Training Objectives

  • LCLAA uses latent regression, contrastive alignment, and action consistency against the frozen expert's action logits; symmetric InfoNCE is applied in both directions.The temperature is fixed, and individual loss components are not ablated.Source: J. Training Objectives · p. 14

K. Latent Robustness and Action-Space Stability

Related papers

Loading 2602.07629v2…