Source-linked AI summary
Channel Gains to Captions: Task-Unified Multi-Level RF Sensing with Vision-Language Models
Tianyu Hu, Zhiren Gong, Haowei Cui, Shuai Wang, Samson Lasaulce, Lingxiang Li, Wassim Hamidouche, Zhi Chen, Merouane Debbah
TL;DR
Existing RF sensing methods use task-specific designs and provide only partial environmental information, limiting support for diverse and unseen 6G sensing requirements. This paper maps mmWave/THz channel gains to multi-level environmental captions with a prompt-conditioned VLM using prompt-routed LoRA experts. Compared with baselines, it achieves superior performance and improves average F1-score by 0.17 over the most competitive variant under an unseen sensing requirement.
Problem
Existing RF sensing methods rely on task-specific designs and provide only partial environmental information, limiting their handling of diverse and unseen 6G sensing requirements.
Method
The framework maps mmWave/THz channel gains to captions through a prompt-conditioned VLM fine-tuned with prompt-routed LoRA experts for level-aware adaptation.
Results
0.17 average F1-score improvement over the most competitive variant is achieved under an unseen sensing requirement, while performance is superior to baselines across multiple semantic levels.
Takeaways & Limitations
Textual prompts support task-unified sensing beyond predefined tasks with broader multi-level semantic coverage.
Abstract
from arXiv · showhide
This letter investigates a task-unified multi-level radio-frequency (RF) sensing framework driven by vision-language models (VLMs). Existing RF sensing methods rely on task-specific designs and provide only partial environmental information, limiting their ability to handle emerging 6G applications. To address this, we propose a generative formulation for RF sensing, where millimeter-wave (mmWave)/terahertz (THz) channel gains are mapped to captions describing multi-level environmental semantics. The framework solves this problem through a complementary design for RF-environment semantic bridging, where a VLM is fine-tuned to leverage its multimodal representations and prompt-conditioned semantic generation capabilities. Hence, different sensing tasks are specified through textual prompts, enabling the framework to handle diverse tasks in a unified manner. For fine-tuning, we introduce prompt-routed low-rank adaptation (LoRA) experts to achieve level-aware adaptation. Simulation results show that, compared with baselines, our framework achieves superior performance with a broader semantic scope, and enables task-unified sensing beyond predefined tasks. Under an unseen sensing requirement, it achieves an average F1-score improvement of 0.17 over the most competitive variant.
I. INTRODUCTION
Existing RF sensing methods use task-specific formulations and capture only partial environmental information, limiting support for diverse and unseen 6G sensing requirements. The proposed framework uses VLM-driven caption generation and prompt-routed LoRA experts to unify multi-level RF sensing.
- Task-specific RF sensing methods estimate obstacle attributes or reconstruct physical environments, but may not provide comprehensive semantics for emerging 6G applications.
- Textual prompts specify sensing requirements, allowing one VLM-based framework to generate task-specific environmental semantics.
- The framework casts RF sensing as mapping mmWave/THz channel gains to captions covering layout, obstacle, and communication-related semantics.
- Prompt-routed LoRA experts fine-tune the pretrained VLM for level-aware adaptation while preserving its pretrained priors.
- Under an unseen sensing requirement, the framework improves average F1-score by 0.17 over the most competitive variant.
A. Multi-Level Environmental Semantics
The paper represents environmental information at layout, obstacle, and communication-related levels using discretized semantic attributes. These levels span spatial structure, individual obstacles, and environment-dependent communication information.
- The environment is represented as I = {I_L, I_O, I_C}, covering layout, obstacle, and communication-related semantics with discretized class labels.
- Layout-level semantics: Layout-level semantics describe large-scale spatial properties such as obstacle area and centroid distributions for propagation characterization.
- Obstacle-level semantics: Obstacle-level semantics describe individual obstacles through attributes including count, centroid, shape, and area classes.
- Communication-related-level semantics: Communication-related semantics include environment labels and candidate low-blockage beam-direction classes for different base-station locations.
B. System Model
The system samples sparse channel gains from users distributed across a gridded 2D area while a central directional base station scans multiple beams. These gains are organized into a tensor whose propagation dependence supports multi-level environmental inference.
- The 2D area is partitioned into N_W × N_W grids, with a central mmWave/THz base station and K users placed at unoccupied grid centers.
- Directional base-station beam scanning uses N_D evenly spaced beams, while users receive through omnidirectional antennas.
- The channel impulse response represents mmWave/THz characteristics including molecular absorption and channel sparsity.
- The average channel power gain is obtained by averaging received signal power over the total observation time T.
- Sparse user channel gains form H_I ∈ R^(N_W×N_W×N_D), whose propagation dependence supports inference of layout and obstacle semantics and then communication-related semantics.
III. VLM-DRIVEN RF SENSING FRAMEWORK
The framework captures environmental semantics from channel gains while conditioning generation on textual sensing-task requirements.
- The VLM-driven framework infers environmental semantics I from channel gains H_I conditioned on sensing task requirements.
A. Problem Formulation
The framework formulates RF sensing as prompt-conditioned caption generation, mapping RF-domain channel gains to textual descriptions of multi-level environmental semantics. This generative formulation supports flexible sensing requirements and richer output than task-specific discriminative formulations.
- RF sensing maps channel gains and textual prompts to generated environmental captions.The model produces token probabilities over a vocabulary and decodes the most probable token sequence.
- Textual prompts specify sensing requirements, allowing sensing tasks to vary flexibly.
- The generated output is compositional, supporting flexible caption length, content richness, and semantic levels.
- Existing discriminative formulations remain restricted to specific sensing requirements and fixed output spaces.
- Cross-entropy loss trains the predicted token distributions against ground-truth one-hot caption targets.The target caption is obtained by verbalizing environmental semantics and encoding each token as a one-hot vocabulary vector.
- A level-dependent gap between RF representations and linguistic semantics, together with sparse RF priors and high-dimensional obstacle attributes, challenges the formulation.
B. Framework Architecture
The architecture combines a pre-trained VLM, RF reconstruction, and prompt-routed LoRA experts to generate and parse multi-level environmental captions. Prompt information selects a weighted mixture of level-specific RF adaptations.
- The framework inputs channel gains and a textual prompt to a pre-trained VLM and reconstructs multi-level environmental semantics from the generated caption.
- VLM multimodal representations provide spatial knowledge and allow reconstructed RF gains to be processed as images.
- RF channel gains are reconstructed into a complete tensor and converted into a visual-form input for the VLM.The reconstruction module supplies auxiliary estimates over unmeasured grids under sparse observations.
- Prompt-routed LoRA experts correspond to layout, obstacle, and communication-related semantic levels.
- The prompt embedding is processed by a two-layer network to produce weights that combine the three experts’ parameter updates.The combined update is expressed as ∆Θ = w^T∆Θ′.
- The fine-tuned VLM parameters add the scaled prompt-routed LoRA update to the pre-trained parameters.The scaling factor and LoRA rank control the adaptation update.
C. LoRA-based Fine-Tuning
LoRA-based fine-tuning adapts the pre-trained VLM through low-rank updates organized into level-specific experts. The empirical objective optimizes these updates while preserving the VLM’s autoregressive generation mechanism.
- Each semantic level uses a separate LoRA expert with a low-rank update formed by multiplying matrices B_i and A_i.The rank satisfies r ≪ min(a, b), reducing the update dimensionality.
- The LoRA matrices are optimized using an empirical objective that reformulates the RF captioning problem.
- The expert matrices are stacked into B′ and A′, then mapped to the corresponding parameter updates.
- Constraint (3b) represents the VLM’s autoregressive mechanism during fine-tuning.
IV. EXPERIMENTAL RESULTS
Experiments evaluate multi-level RF sensing across simulated and real-world scenarios, comparing the proposed framework with task-specific, fixed-output, and ablated baselines. The framework generally performs best across semantic levels and shows stronger results for complex and unseen sensing requirements, although performance remains limited in highly complex unseen conditions.
- Experimental setup: 250 S1 scenarios generate seven prompt-defined samples each, while S2 and S3 evaluate models on unseen scenarios after fine-tuning on S1.S1 obstacle counts range from 1 to 5, and S2/S3 serve as unseen validation scenarios.
- Baselines and metrics: Averaged macro-F1 evaluates layout, obstacle, and communication-related semantics, with higher scores indicating better precision and recall.The baselines include task-specific U-NetGAN, fixed-output ResNet-18, and ablated variants of the proposed framework.
- S2 results: The proposed framework generally achieves superior performance over all baselines across layout, obstacle, and communication-related semantic levels in S2.Its obstacle-attribute and communication-related sensing capabilities remain only moderate.
- S3 results: For S3, the proposed framework achieves higher F1 scores than baselines when the number of obstacles exceeds two.The results are reported as averaged macro-F1 scores versus obstacle count.
- S3 results: Increasing-then-decreasing performance with obstacle count suggests that additional sensing cues eventually saturate, while overly complex environments become more difficult.This trend is observed for the proposed framework and its ablated variants.
- Unseen-task results: 0.17 average F1 improvement over “Ablated: w/o routing” is achieved on an unseen beam-direction sensing requirement in S3.The proposed framework also reaches only 0.09 F1 in the eight-obstacle case.
V. CONCLUSION
The paper proposes a VLM-adapted framework for task-unified multi-level RF sensing from mmWave/THz channel gains. Simulations report superior performance over baselines and generalization to diverse sensing requirements.
- Conclusion: The proposed framework adapts VLMs for task-unified multi-level RF sensing from mmWave/THz channel gains.It generates captions spanning layout, obstacle, and communication-related environmental semantics.
- Conclusion: Simulation results show superior RF sensing performance over baselines and generalization to diverse sensing requirements.The framework processes diverse sensing requirements and generates environmental semantic captions.