Source-linked AI summary
Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents
Xi Chen, Zhe Liu, Xiaogang Xu, Jiafei Xu, Chunyi Zhou, Yuan Su, Rui Zeng, Tianyu Du, Kelu Yao, Chao Li, Shouling Ji
TL;DR
Aerial VLN agents are usually trained on detailed, trajectory-aligned commands, while users give shorter, intent-driven instructions. TGIT keeps the navigator frozen and learns agent-executable translations from trajectory outcomes. It improves OpenFly SR and transfers zero-shot to Human instructions, while its activation remains bounded by the navigator's capabilities.
Problem
Aerial VLN agents are typically trained on detail-rich, trajectory-aligned instructions, whereas users issue shorter, intent-driven commands, creating an instruction-interface gap.
Source: §1 · p. 1Method
TGIT keeps the UAV policy frozen and adapts a LoRA VLM front-end using hard cases, executed candidate translations, and trajectory-outcome preferences.
Source: §1 · p. 2Results
TGIT raises OpenFly SR from 15.27% to 37.93% and transfers zero-shot to raise Human-input SR from 11.33% to 32.51%, with gains also reported on held-out OpenFly and other frozen agents.
Source: §1 · p. 2, §4.2 · p. 5Takeaways & Limitations
TGIT repairs recoverable instruction-interface failures by activating demonstrated navigation skill, but translation cannot strengthen the frozen backbone.
Source: §5 · p. 7TGIT performs one-shot translation before navigation and does not revise commands online as observations or execution states change.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this \emph{instruction gap} drops success rate (SR) from $31.03\%$ to $11.33\%$. To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield $15.27\%$ SR. We introduce the \textbf{Trajectory-Grounded Instruction Translator (TGIT)}, a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes. The resulting Weak-trained translator raises Weak-input SR to $37.93\%$ and transfers zero-shot to real human instructions ($11.33\%{\rightarrow}32.51\%$); it also improves held-out OpenFly ($4.95\%{\rightarrow}20.79\%$) and yields recovery on CityNav and AirVLN.
1 Introduction
Aerial VLN agents are typically trained on detail-rich, trajectory-aligned commands, while users give shorter, intent-driven instructions; on a frozen navigator, this mismatch lowers success. TGIT keeps the navigator fixed and learns translations from trajectory outcomes, improving OpenFly performance and transferring zero-shot to Human instructions.
1 Introduction
- With routes, initial conditions, and the navigator fixed, SR falls from 31.03% with dataset instructions to 11.33% with Human instructions, exposing a training–deployment language mismatch.Users tend to issue shorter, intent-driven commands than the detail-dense, trajectory-aligned instructions used to train aerial VLN agents.Source: §1 · p. 1
- Text rewriting alone cannot establish whether a particular navigator will succeed, while retraining can change perception and control and complicate attribution of gains.TGIT instead keeps the navigator fixed and learns agent-executable translations from trajectory outcomes.Source: §1 · pp. 1–2
- Because Human annotation is costly to scale, the authors use a language model with Human-written style examples to turn Orig commands into shorter, intent-centered Weak counterparts.Translator supervision focuses on paired cases where Orig succeeds but Weak fails under the same route, initial condition, simulator, and policy.Source: §1 · p. 2
- TGIT freezes the UAV policy and adapts a LoRA VLM front-end, using hard cases, executed candidate translations, trajectory-ranked preferences, and multi-round DPO with pool resampling.The translator is evaluated on Weak inputs and transferred zero-shot to Human inputs.Source: §1 · p. 2
- On the 203-episode OpenFly evaluation, TGIT raises SR from 15.27% to 37.93%, exceeding Orig (31.03%), ICL, and SFT, while Human-input SR rises zero-shot from 11.33% to 32.51%.Held-out OpenFly scenes and two additional frozen agents also show relative activation, while absolute success remains bounded by the navigator.Source: §1 · p. 2
2 Related Work
Prior aerial VLN work improves navigation capability, while instruction-centric work generates or rewrites language for clarity, specificity, or coverage. TGIT instead keeps the aerial policy frozen and learns an interface intended to activate competence already present in that policy.
Our position
- Prior VLN and embodied-language work is framed around two axes—improving the navigator and rewriting or generating language—whereas TGIT keeps the aerial policy frozen and targets interface activation.The stated goal is to activate competence already present in the policy.Source: §2 · p. 2
Agent-centric aerial VLN
- Agent-centric aerial VLN advances continuous control, long-horizon navigation, and sim-to-real evaluation, typically using detailed, trajectory-aligned training instructions.The passage traces the field from indoor VLN to continuous outdoor UAV flight and larger aerial environments.Source: §2 · p. 2
Instruction-centric language interfaces
- Instruction-centric work scales synthetic supervision and develops landmark-centered, decomposed, or rewritten instructions, optimizing clarity, recognizability, or training-data coverage.Examples include VLN-Trans (Zhang and Kordjamshidi 2023), T-araVLN (Zhao et al. 2025), and RAM (Wei et al. 2025).Source: §2 · p. 2
Our position
- TGIT ranks candidate translations with simulator rollouts and updates only a LoRA front-end, using Orig-success/Weak-failure cases to target interface activation rather than policy retraining or instruction imitation.The passage distinguishes this approach from multimodal preference methods that align VLM responses or retrain the VLA itself.Source: §2 · p. 2
3 Methodology
TGIT adapts an instruction front end while keeping the UAV navigator fixed, using trajectory outcomes to align weakened inputs with executable commands. Its training loop selects recoverable failures, ranks candidate rewrites through frozen rollouts, and updates the translator with preferences; deployment uses a single forward pass.
3.1 Problem Formulation
- TGIT freezes the navigator and controller, trains only the translator, and uses available RGB context alongside Human-informed weakened inputs rather than treating them as an estimator of the Human distribution.The translator emits a command for the unchanged navigator; rewards, actions, goal labels, and policy updates are unavailable to it at deployment.Source: §3.1 · p. 3
3.2 Trajectory-Grounded Instruction Translation
- The hard-case pool retains episodes where the original instruction succeeds but its paired weakened instruction fails under the same route, initial condition, simulator, and policy.This selection targets cases where the navigator has demonstrated the required capability, rather than failures under both instructions.Source: §3.2 · p. 4
- For each sampled weakened input, the translator proposes three candidates conditioned on visual context, and the frozen navigator executes each to produce trajectory outcomes.The original dataset instruction is also included during training as a strong execution reference; inference uses only the human-style input and available observations.Source: §3.2 · p. 4
- Candidate trajectories are ranked with a weighted reward using success, path efficiency, and normalized final navigation error, then converted into pairwise preferences.The protocol weights these terms 0.5, 0.3, and 0.2, respectively, with λ = 0.1; near-tied outcomes are discarded to avoid simulator-noise supervision.Source: §3.2 · p. 4
- Multi-round DPO updates only LoRA translator parameters, chains adapters and resamples hard cases; deployment instead emits one command without rewards, preference pairs, or policy updates.Each round samples K = 20 hard cases, generates and executes candidates, and updates the translator; the UAV policy remains unchanged.Source: §3.2 · pp. 3–4
4 Experiments
Across matched evaluations with frozen UAV agents, TGIT improves success on Weak inputs and transfers across scenes and agents. Ablations examine how hard-case selection, rollout ranking, and translator backbone affect that recovery.
4.1 Experimental Setup
- The protocol evaluates three frozen UAV agents across Selected, Unselected, and Unseen scenes, with matched conditions and zero-shot Human transfer on OpenFly Selected-Seen.Reported measures are SR, success-conditional SPL, and NE.Source: §4.1 · pp. 4–5
4.2 Results
- On the same 203 OpenFly episodes and frozen navigator, SR is 31.03% with Orig, 15.27% with Weak, and 11.33% with Human instructions.Human and TGIT(H) evaluation is limited to OpenFly Selected-Seen; TGIT(H) reuses the Weak-trained adapter zero-shot.Source: §4.2 · p. 5, §4.1 · p. 5
- TGIT improves OpenFly Unselected-Seen SR from 14.11% to 25.25% and Unseen SR from 4.95% to 20.79%; CityNav and AirVLN also improve, with low absolute success.AirVLN methods remain at 0.00% SR on Unseen.Source: §4.2 · p. 5
4.3 Ablation Study
- Hard-case selection plus resampling K=20 from a 100-case pool improves through Round 5 to the TGIT result; fixed-subset training saturates and random sampling is unstable.Table 2 reports the Round 5 result as 37.93 / 24.85 / 72.08 for SR (%) / SPL (%) / NE (m).Source: §4.3 · pp. 5–6, Table 2 · p. 7
- Weak failures are dominated by early collapse (153/203); TGIT expands success to 37.93% and shrinks early failure, while frame diagnostics approach its result.Figure 3 presents pool curves, failure stages, and f1*/f4* diagnostics; its TGIT early/late proportions use a renormalized Selected-Pool diagnostic.Source: §4.3 · p. 6, Figure 3 · p. 7
- On a fixed Round 5 candidate set, Success-only reaches 35.47% SR and lexicographic Success→SPL→NE reaches 36.14%, supporting robustness beyond the Weighted ranking.Weighted remains the main protocol because it retains dense efficiency and distance signals for unsuccessful or near-tied candidates.Source: §4.3 · p. 6, Table 3 · p. 7
- Under the diagnostic N=100, M=4 Round 5 protocol, Qwen2.5-VL-7B reaches 36.45% SR with full valid-output coverage; the comparison is not a monotonic scaling test.The 2B model has 156/203 valid outputs, making its absolute SR unreliable under the fixed denominator.Source: §4.3 · p. 6
5 Conclusion
TGIT addresses recoverable instruction-interface failures while keeping the UAV policy frozen, improving OpenFly Selected-Seen success. Its reach remains bounded by navigator capability and one-shot translation.
5 Conclusion
- OpenFly Selected-Seen SR rises from 15.27% to 37.93%, exceeding Orig (31.03%), ICL, and SFT.Table 4's backbone/scale comparison is diagnostic; Table 1 remains the primary result.Source: §5 · p. 7, Table 4 · p. 7
- On CityNav and AirVLN, translation activates demonstrated skill but cannot strengthen the navigation backbone.Source: §5 · p. 7
- TGIT translates once before navigation and does not revise commands as observations or execution states change.The agents also remain limited in perception and control.Source: §5 · p. 7