PaperlayerView exact revision on arXiv

Source-linked AI summary

Channel Gains to Captions: Task-Unified Multi-Level RF Sensing with Vision-Language Models

Tianyu Hu, Zhiren Gong, Haowei Cui, Shuai Wang, Samson Lasaulce, Lingxiang Li, Wassim Hamidouche, Zhi Chen, Merouane Debbah

arXiv:2608.30836v1eess.SP

TL;DR

Existing RF sensing methods use task-specific designs and provide only partial environmental information, limiting support for diverse and unseen 6G sensing requirements. This paper maps mmWave/THz channel gains to multi-level environmental captions with a prompt-conditioned VLM using prompt-routed LoRA experts. Compared with baselines, it achieves superior performance and improves average F1-score by 0.17 over the most competitive variant under an unseen sensing requirement.

  • Problem

    Existing RF sensing methods rely on task-specific designs and provide only partial environmental information, limiting their handling of diverse and unseen 6G sensing requirements.

    Source: §I · p. 1

  • Method

    The framework maps mmWave/THz channel gains to captions through a prompt-conditioned VLM fine-tuned with prompt-routed LoRA experts for level-aware adaptation.

    Source: §I · p. 1, C. LoRA-based Fine-Tuning · p. 4

  • Results

    0.17 average F1-score improvement over the most competitive variant is achieved under an unseen sensing requirement, while performance is superior to baselines across multiple semantic levels.

    Source: §IV · pp. 4–5

  • Takeaways & Limitations

    Textual prompts support task-unified sensing beyond predefined tasks with broader multi-level semantic coverage.

    Source: §I · p. 1, §V · p. 5

Abstract

from arXiv · show

This letter investigates a task-unified multi-level radio-frequency (RF) sensing framework driven by vision-language models (VLMs). Existing RF sensing methods rely on task-specific designs and provide only partial environmental information, limiting their ability to handle emerging 6G applications. To address this, we propose a generative formulation for RF sensing, where millimeter-wave (mmWave)/terahertz (THz) channel gains are mapped to captions describing multi-level environmental semantics. The framework solves this problem through a complementary design for RF-environment semantic bridging, where a VLM is fine-tuned to leverage its multimodal representations and prompt-conditioned semantic generation capabilities. Hence, different sensing tasks are specified through textual prompts, enabling the framework to handle diverse tasks in a unified manner. For fine-tuning, we introduce prompt-routed low-rank adaptation (LoRA) experts to achieve level-aware adaptation. Simulation results show that, compared with baselines, our framework achieves superior performance with a broader semantic scope, and enables task-unified sensing beyond predefined tasks. Under an unseen sensing requirement, it achieves an average F1-score improvement of 0.17 over the most competitive variant.

I. INTRODUCTION

Existing RF sensing methods use task-specific formulations and capture only partial environmental information, limiting support for diverse and unseen 6G sensing requirements. The proposed framework uses VLM-driven caption generation and prompt-routed LoRA experts to unify multi-level RF sensing.

  • Task-specific RF sensing methods estimate obstacle attributes or reconstruct physical environments, but may not provide comprehensive semantics for emerging 6G applications.Source: §I · p. 1
  • Textual prompts specify sensing requirements, allowing one VLM-based framework to generate task-specific environmental semantics.Source: §I · p. 1
  • The framework casts RF sensing as mapping mmWave/THz channel gains to captions covering layout, obstacle, and communication-related semantics.Source: §I · p. 1
  • Prompt-routed LoRA experts fine-tune the pretrained VLM for level-aware adaptation while preserving its pretrained priors.Source: §I · p. 1
  • Under an unseen sensing requirement, the framework improves average F1-score by 0.17 over the most competitive variant.Source: §I · p. 1

A. Multi-Level Environmental Semantics

The paper represents environmental information at layout, obstacle, and communication-related levels using discretized semantic attributes. These levels span spatial structure, individual obstacles, and environment-dependent communication information.

Layout-level semantics

Obstacle-level semantics

Communication-related-level semantics

B. System Model

The system samples sparse channel gains from users distributed across a gridded 2D area while a central directional base station scans multiple beams. These gains are organized into a tensor whose propagation dependence supports multi-level environmental inference.

  • The 2D area is partitioned into N_W × N_W grids, with a central mmWave/THz base station and K users placed at unoccupied grid centers.Source: B. System Model · p. 2
  • Directional base-station beam scanning uses N_D evenly spaced beams, while users receive through omnidirectional antennas.Source: B. System Model · p. 2
  • The channel impulse response represents mmWave/THz characteristics including molecular absorption and channel sparsity.Source: B. System Model · p. 2
  • The average channel power gain is obtained by averaging received signal power over the total observation time T.Source: B. System Model · p. 2
  • Sparse user channel gains form H_I ∈ R^(N_W×N_W×N_D), whose propagation dependence supports inference of layout and obstacle semantics and then communication-related semantics.Source: B. System Model · p. 2

III. VLM-DRIVEN RF SENSING FRAMEWORK

The framework captures environmental semantics from channel gains while conditioning generation on textual sensing-task requirements.

  • The VLM-driven framework infers environmental semantics I from channel gains H_I conditioned on sensing task requirements.Source: §III · p. 2

A. Problem Formulation

The framework formulates RF sensing as prompt-conditioned caption generation, mapping RF-domain channel gains to textual descriptions of multi-level environmental semantics. This generative formulation supports flexible sensing requirements and richer output than task-specific discriminative formulations.

  • RF sensing maps channel gains and textual prompts to generated environmental captions.The model produces token probabilities over a vocabulary and decodes the most probable token sequence.Source: A. Problem Formulation · p. 2
  • Textual prompts specify sensing requirements, allowing sensing tasks to vary flexibly.Source: A. Problem Formulation · p. 2
  • The generated output is compositional, supporting flexible caption length, content richness, and semantic levels.Source: A. Problem Formulation · p. 2
  • Existing discriminative formulations remain restricted to specific sensing requirements and fixed output spaces.Source: A. Problem Formulation · p. 2
  • Cross-entropy loss trains the predicted token distributions against ground-truth one-hot caption targets.The target caption is obtained by verbalizing environmental semantics and encoding each token as a one-hot vocabulary vector.Source: A. Problem Formulation · pp. 2–3
  • A level-dependent gap between RF representations and linguistic semantics, together with sparse RF priors and high-dimensional obstacle attributes, challenges the formulation.Source: A. Problem Formulation · p. 3

B. Framework Architecture

The architecture combines a pre-trained VLM, RF reconstruction, and prompt-routed LoRA experts to generate and parse multi-level environmental captions. Prompt information selects a weighted mixture of level-specific RF adaptations.

  • The framework inputs channel gains and a textual prompt to a pre-trained VLM and reconstructs multi-level environmental semantics from the generated caption.Source: B. Framework Architecture · p. 3
  • VLM multimodal representations provide spatial knowledge and allow reconstructed RF gains to be processed as images.Source: B. Framework Architecture · p. 3
  • RF channel gains are reconstructed into a complete tensor and converted into a visual-form input for the VLM.The reconstruction module supplies auxiliary estimates over unmeasured grids under sparse observations.Source: B. Framework Architecture · p. 3
  • Prompt-routed LoRA experts correspond to layout, obstacle, and communication-related semantic levels.Source: B. Framework Architecture · p. 3
  • The prompt embedding is processed by a two-layer network to produce weights that combine the three experts’ parameter updates.The combined update is expressed as ∆Θ = w^T∆Θ′.Source: B. Framework Architecture · p. 3
  • The fine-tuned VLM parameters add the scaled prompt-routed LoRA update to the pre-trained parameters.The scaling factor and LoRA rank control the adaptation update.Source: B. Framework Architecture · pp. 3–4

C. LoRA-based Fine-Tuning

LoRA-based fine-tuning adapts the pre-trained VLM through low-rank updates organized into level-specific experts. The empirical objective optimizes these updates while preserving the VLM’s autoregressive generation mechanism.

IV. EXPERIMENTAL RESULTS

Experiments evaluate multi-level RF sensing across simulated and real-world scenarios, comparing the proposed framework with task-specific, fixed-output, and ablated baselines. The framework generally performs best across semantic levels and shows stronger results for complex and unseen sensing requirements, although performance remains limited in highly complex unseen conditions.

Experimental setup

  • 250 S1 scenarios generate seven prompt-defined samples each, while S2 and S3 evaluate models on unseen scenarios after fine-tuning on S1.S1 obstacle counts range from 1 to 5, and S2/S3 serve as unseen validation scenarios.Source: §IV · p. 4

Baselines and metrics

  • Averaged macro-F1 evaluates layout, obstacle, and communication-related semantics, with higher scores indicating better precision and recall.The baselines include task-specific U-NetGAN, fixed-output ResNet-18, and ablated variants of the proposed framework.Source: §IV · p. 4

S2 results

  • The proposed framework generally achieves superior performance over all baselines across layout, obstacle, and communication-related semantic levels in S2.Its obstacle-attribute and communication-related sensing capabilities remain only moderate.Source: §IV · p. 4, Figure 3 · p. 4

S3 results

  • For S3, the proposed framework achieves higher F1 scores than baselines when the number of obstacles exceeds two.The results are reported as averaged macro-F1 scores versus obstacle count.Source: §IV · p. 4, Figure 4 · p. 5
  • Increasing-then-decreasing performance with obstacle count suggests that additional sensing cues eventually saturate, while overly complex environments become more difficult.This trend is observed for the proposed framework and its ablated variants.Source: §IV · p. 5

Unseen-task results

  • 0.17 average F1 improvement over “Ablated: w/o routing” is achieved on an unseen beam-direction sensing requirement in S3.The proposed framework also reaches only 0.09 F1 in the eight-obstacle case.Source: §IV · p. 5, Table II · p. 5

V. CONCLUSION

The paper proposes a VLM-adapted framework for task-unified multi-level RF sensing from mmWave/THz channel gains. Simulations report superior performance over baselines and generalization to diverse sensing requirements.

Conclusion

  • The proposed framework adapts VLMs for task-unified multi-level RF sensing from mmWave/THz channel gains.It generates captions spanning layout, obstacle, and communication-related environmental semantics.Source: §V · p. 5
  • Simulation results show superior RF sensing performance over baselines and generalization to diverse sensing requirements.The framework processes diverse sensing requirements and generates environmental semantic captions.Source: §V · p. 5
Loading 2608.30836v1…