Source-linked AI summary
FRESHLATENT: Channel-Aware Latent Adaptation for Resource-Constrained Embodied VLM Perception
Rajat Bhattacharjya, Minwoo Kim, Arnab Sarkar, Tamoghno Das, Sing-Yao Wu, Eli Bozorgzadeh, Marco Levorato, Nikil Dutt
TL;DR
Mission-guided UAV perception can use split VLMs, but transmitting intermediate features exposes downstream segmentation to wireless corruption. FreshLatent adapts the latent interface while the surrounding VLM stays frozen. At 0 dB, it recovers 63.5–69.1% of the gIoU improvement of a heavier codec across three budgets and uses substantially less embedded interface cost.
Problem
Mission-guided UAV perception needs reliable query-conditioned spatial evidence, but wireless corruption of transmitted split-VLM representations can degrade segmentation while VLM compute and memory challenge onboard deployment.
Source: §1 · p. 1Method
FreshLatent adapts a frozen split-VLM interface by training only its latent encoder and decoder through channel-aware reconstruction of the original pretrained feature.
Source: §1 · p. 2, §3.2 · p. 3Results
At 0 dB, FreshLatent recovers 63.5–69.1% of the gIoU improvement achieved by FFMNet-JSCC-Range across three communication budgets.
Source: §4.4 · p. 4Takeaways & Limitations
FreshLatent broadens the quality-valid channel–communication region, satisfying mission-conditioned perception requirements across more constrained operating conditions.
Source: §4.4 · p. 4, §5 · p. 5The minimum acceptable perception-quality threshold is not universal; evaluation uses a range of representative thresholds.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Mission-critical UAVs increasingly rely on split vision-language model (VLM) perception under tight onboard-resource and wireless-communication constraints. However, corruption of transmitted intermediate features creates a deployment mismatch for clean-trained split interfaces, while stronger channel-aware codecs can impose substantial onboard cost. We present FreshLatent, a lightweight channel-aware latent adapter that trains a power-normalized encoder-decoder through wireless corruption while keeping the surrounding VLM frozen. We formulate deployment around a mission-conditioned perception requirement and embedded interface cost, linking channel quality and communication budget to the operating conditions under which perception remains usable. At 0 dB and the tightest communication budget, FreshLatent improves gIoU and cIoU over clean split compression by 20.79 and 20.87 points, respectively. At the most adverse evaluated SNR (0 dB), across all three communication budgets, FreshLatent recovers 63.5-69.1% of the gIoU improvement achieved by a much heavier, range-trained feature-JSCC codec. On an NVIDIA Jetson AGX Xavier in 10-W mode, FreshLatent uses 37-40x fewer encoder parameters, 7.7-9.9x lower edge-interface latency, and 8.8-10.0x lower edge-interface energy than the heavier codec. Together, these results show that lightweight channel-aware adaptation can recover a substantial fraction of the robustness of a much larger communication interface while broadening quality-valid operation under constrained wireless conditions.
1. INTRODUCTION
FreshLatent adapts a lightweight split-VLM interface to wireless corruption while keeping the surrounding VLM frozen. The paper frames this design around mission-conditioned perception requirements and embedded resource feasibility.
1. INTRODUCTION
- Mission-critical UAVs need query-conditioned spatial evidence for operator-guided decisions, but VLM resource demands motivate split processing and wireless feature transfer can degrade downstream segmentation.The transmitted representation must remain compatible with the downstream VLM.Source: §1 · p. 1
- The open question is how to make a lightweight split-VLM interface robust to wireless corruption without substantially increasing edge overhead.More expressive channel-aware codecs may add capacity and computation, complicating attribution of robustness gains.Source: §1 · p. 1
- FreshLatent trains a power-normalized latent encoder–decoder through wireless corruption to reconstruct pretrained features while leaving the surrounding VLM unchanged.This separates adaptation to the channel from additional codec capacity.Source: §1 · pp. 1–2
- The paper formulates mission-conditioned perception requirements and embedded resource feasibility, then presents FreshLatent as a lightweight channel-aware interface that recovers a substantial fraction of a heavier codec’s poor-channel robustness gain.The comparison is under matched range training.Source: §1 · p. 2
2. SYSTEM MODEL AND PROBLEM FORMULATION
The system splits query-conditioned perception between a resource-constrained UAV and a remote server, with the wireless link carrying an intermediate VLM representation. Task validity combines a mission-specific minimum perception quality with feasible onboard interface cost.
2.1. Operator-Guided Split VLM Perception
- At latent rate r, a rate-specific encoder compresses the split feature, and a server-side decoder reconstructs it after transmission through a channel characterized by SNR γ and realization ω.A separate encoder–decoder pair is used for each supported latent rate.Source: §2.1 · p. 2
- The UAV sends an intermediate representation to a remote server, where the frozen VLM combines its reconstruction with the operator query and auxiliary representation to produce query-conditioned segmentation.The operator receives the segmentation as spatial evidence for mission-level decisions.Source: §2.1 · p. 2, §2 · p. 2
- Unlike image offloading, the wireless link carries an internal model representation, so channel distortion can alter the reconstructed feature and propagate through the frozen downstream computation.The system compares architectures using a common communication-budget variable K, measured in channel uses.Source: §2.1 · p. 2
2.2. Mission-Conditioned Task Validity
- Mission context is represented by a semantic utility that marks perception quality as acceptable when it meets a minimum requirement.The formulation evaluates representative thresholds rather than assigning one universal value.Source: §2.2 · p. 2
- Onboard resource feasibility is assessed through an interface-cost vector that can include parameter footprint, codec latency, and codec energy.The cost components must satisfy the corresponding deployment resource limits.Source: §2.2 · p. 2
- A task-valid operating point requires both sufficient perception quality and feasible onboard interface cost.A deployment can combine the measured quality and cost with its own resource budget to identify valid operating points.Source: §2.2 · p. 2
3. FRESHLATENT: CHANNEL-AWARE LATENT ADAPTATION
FreshLatent adapts a lightweight split-VLM interface to wireless corruption while keeping the surrounding model frozen. Its spatially preserving bottleneck compresses intermediate features, and channel-aware training reconstructs the pretrained features expected downstream.
3.1. Lightweight Latent Interface
- The latent interface applies rate-specific linear projections along the channel dimension, reducing feature width while preserving the H × W spatial organization.Each supported rate has a separate lightweight bottleneck; the server-side decoder restores the feature dimensionality expected by the frozen post-split computation.Source: §3.1 · p. 3
3.2. Channel-Aware Reconstruction Learning
- FreshLatent places wireless corruption in the reconstruction path during training to address the mismatch faced by clean-trained decoders at deployment.The original pretrained feature F remains the reconstruction target, keeping the reconstructed output compatible with the frozen downstream VLM.Source: §3.2 · p. 3
- Only the latent encoder and decoder are optimized through the channel-aware path; the feature extractor and downstream VLM remain frozen.Training uses γtr and a sampled channel realization ω.Source: §3.2 · p. 3
- One range-trained FreshLatent checkpoint per latent rate is evaluated across channel conditions and compared with SNR-Specialist checkpoints selected by evaluation SNR.FL-Fixed instead concentrates training at 13 dB.Source: §3.2 · p. 3
4. EXPERIMENTS AND EVALUATION
The evaluation compares split-VLM perception, mission-conditioned quality coverage, and embedded interface costs across matched communication budgets and channel conditions. FreshLatent recovers a substantial share of the heavier range-trained codec’s poor-channel gIoU gain while expanding quality-valid operation at lower interface cost.
4.1. Experimental Setup
- The evaluation uses LISA-7B with SAM ViT-H, ReasonSeg validation, three matched channel-use budgets, AWGN conditions, and Jetson AGX Xavier profiling in MODE_10W.FreshLatent and FFMNet-JSCC-Range are trained over U[0, 18] dB; the AWGN-trained FreshLatent is also evaluated under unseen Rayleigh fading at r = 0.10.Source: §4.1 · p. 3
4.2. Matched-Budget Perception Robustness
- 20.79 gIoU points and 20.87 cIoU points are FreshLatent’s gains over Clean at 0 dB and K = 0.262 × 10^6 channel uses per image.At K = 1.311 × 10^6, the 0-dB gIoU gain is 8.05 points; the gap narrows as the communication budget increases.Source: §4.2 · p. 3, Figure 3 · p. 4
- Across 27 AWGN operating points, range-trained FreshLatent differs from SNR-Specialist by 0.36 gIoU and 0.57 cIoU points on average, and scores higher in gIoU at 8 points.Under unseen Rayleigh fading, the AWGN-trained model has average degradation of 7.28 gIoU and 8.92 cIoU points over 0–25 dB.Source: §4.2 · p. 3
- At 0 dB, FreshLatent recovers 63.5–69.1% of the gIoU improvement achieved by FFMNet-JSCC-Range over Clean across all three budgets.This comparison uses matched range training; FFMNet-JSCC-Range exceeds fixed-13 dB FFMNet-JSCC at 8 of 9 points below 9 dB and ties at the remaining point.Source: §4.2 · pp. 3–4
4.3. Mission-Conditioned Quality-Valid Region
- FreshLatent expands quality-valid coverage over Clean by 3.7–11.1 percentage points across the shown gIoU requirements.At τµ = 0.72 and 15 dB, it meets the requirement with K = 0.524 × 10^6 rather than 1.311 × 10^6 channel uses, reducing the minimum tested budget by 60%.Source: §4.3 · p. 4, Figure 4 · p. 4
4.4. Embedded Interface Cost
- Across three budgets, FreshLatent uses 37–40× fewer encoder parameters, 7.7–9.9× lower interface latency, and 8.8–10.0× lower interface energy than FFMNet-JSCC.Fixed- and range-trained variants share the same architecture and embedded interface cost.Source: §4.4 · p. 4
- Clean achieves greater quality-valid coverage than DeepJSCC-Image for every evaluated requirement except τµ = 0.60.Source: §4.4 · p. 4
5. CONCLUSION AND FUTURE WORK
Under matched range training, FreshLatent recovers a substantial fraction of a heavier feature-JSCC codec’s poor-channel robustness gain while expanding the quality-valid channel–communication region over clean split inference. It does so with substantially lower embedded interface cost; future work will examine runtime control, selective fallback, and operator-level evaluation.
5. CONCLUSION AND FUTURE WORK
- FreshLatent recovers a substantial fraction of a heavier feature-JSCC codec’s poor-channel robustness gain while expanding the quality-valid channel–communication region over clean split inference.The comparison uses matched range training.Source: §5 · pp. 4–5
- On an NVIDIA Jetson AGX Xavier, FreshLatent uses 37–40× fewer encoder parameters, 7.7–9.9× lower interface latency, and 8.8–10.0× lower interface energy than the heavier same-split alternative.Source: §5 · p. 5
- Future work will examine runtime communication–computation control under changing conditions, selective fallback when communication-dependent reasoning becomes unreliable, and operator-level evaluation.Source: §5 · p. 5