Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding

Jeongwan Shin; Jaehyeon Kim; Donguk Ko; Jaeho Choi

Source-linked AI summary

Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding

Jeongwan Shin, Jaehyeon Kim, Donguk Ko, Jaeho Choi

arXiv:2608.14179v1Aug 14, 2026cs.AI

TL;DR

mmWave-LLM integration lacks paired language data, standardized evaluation, and foundational encoders. The paper introduces mmWave-QA and textualizes point clouds for off-the-shelf LLM question answering, finding greater robustness than RGB under low light and occlusion.

Problem
mmWave-LLM integration lacks public radar-language pairs, standardized encoders, and cross-dataset consistency despite mmWave’s resilience to low light and occlusion.
Method
The paper serializes 5D mmWave point clouds as coordinate-formatted text and constructs mmWave-QA by harmonizing heterogeneous datasets into natural-language QA tasks.
Results
LLMs effectively interpret textualized mmWave inputs, with greater robustness than RGB under visually degraded conditions, especially low light and occlusion.
Takeaways & Limitations
mmWave-QA provides a unified testbed for studying language-conditioned radar human understanding across diverse devices, environments, scenarios, and QA tasks.
Takeaways & Limitations
Evaluation with diverse open-source LLMs remains limited, motivating future work on improving open-source models through fine-tuning.

Abstract

from arXiv · show

Large language models (LLMs) have shown remarkable reasoning and generative capabilities, motivating their use as universal reasoning engines for perception. While modern approaches such as vision-language models (VLMs) have attempted to incorporate reasoning capabilities into visual sensing, the integration of LLMs with the millimeter-wave (mmWave) modality-despite its unique advantages under low light and occlusion-remains largely unexplored. The principal bottlenecks stem from the scarcity of radar language pairs, severe cross-dataset heterogeneity, and the absence of a foundational mmWave encoder. We address this gap through a minimal textualization interface that serializes each mmWave point cloud into concise natural language, allowing off-the-shelf LLMs to operate in a question answering (QA) setting. Building on this, we present mmWave-QA, the first benchmark for language-conditioned mmWave human perception. mmWave-QA aggregates heterogeneous public mmWave datasets and harmonizes them via calibration-aware preprocessing and global taxonomy alignment, while providing natural language QA. Spanning six scenarios and five QA tasks, the benchmark enables standardized evaluation across diverse mmWave hardware and experimental conditions, establishing a foundation for scalable research on mmWave-LLM integration. We further evaluate and analyze LLMs on our mmWave-QA, highlighting their zero-shot reasoning potential for radar perception, as well as their robustness under visual degradation.

1. Introduction

The paper introduces mmWave-QA, a unified natural-language benchmark that enables off-the-shelf LLMs to reason over textualized mmWave point clouds across heterogeneous sensing conditions. It demonstrates zero-shot radar reasoning and highlights mmWave’s robustness and privacy advantages when visual perception degrades.

Motivation: mmWave sensing remains effective under low or mixed lighting, body occlusion, and long standoff distances while preserving facial privacy.It measures human targets through electromagnetic reflections rather than appearance cues and does not reveal facial identity or fine appearance.
Challenges: LLM integration with mmWave is hindered by scarce radar-language pairings and severe heterogeneity across datasets and sensing conditions.Existing mmWave datasets are small, lab-specific, difficult to annotate, and lack public radar-caption pairings.
Approach: Textualizing each 5D point cloud as [x, y, z, Doppler, intensity] enables off-the-shelf LLMs to perform reasonable zero-shot reasoning about human motion.The approach prompts LLMs to answer motion questions without mmWave-specific tuning.
Benchmark: mmWave-QA is the first benchmark to harmonize heterogeneous datasets, align them with a common taxonomy, and generate reviewed natural-language QA pairs.Its curation pipeline combines LLM–human QA generation with quality review and refinement.
Benchmark: mmWave-QA spans six scenarios and five complementary QA tasks across different hardware devices and experimental conditions.The tasks probe spatial, temporal, and joint-level aspects of human motion, enabling standardized evaluation under varied subject placements and environments.
Findings: The study reports strong zero-shot reasoning from simple LLMs on textualized radar inputs and greater robustness than RGB under visual degradation.These findings establish mmWave-QA as a testbed for evaluating mmWave-LLM integration.

2. Related Work

Prior mmWave human-sensing research has expanded from action recognition to pose estimation and diverse sensing conditions, but heterogeneous data sources and setups hinder universally adaptable models. Existing multimodal LLM efforts remain limited for mmWave and face generalization or modality-specific encoder constraints, motivating textualized point-cloud reasoning and benchmarking.

mmWave Human Sensing Datasets: mmWave datasets have progressed from basic actions to composite activities, diverse environments, and joint-level pose understanding.The surveyed datasets span action recognition and pose estimation, including rehabilitation and natural daily motions.
mmWave Human Sensing Datasets: Growing heterogeneity in data sources and sensing setups hinders the development of universally adaptable mmWave models.The passage attributes this limitation to diversity in sensing conditions and data sources.
mmWave-based Human Understanding: mmWave human-understanding methods use CNNs for spatial representations, LSTMs for temporal dependencies, and transformers for broader context understanding.The surveyed paradigms include MARS and mmPose, RadHAR and DP-CBL, and transformer-inspired approaches.
Sensing modalities with LLMs: Although LLMs support image, video, audio, and spatial-measurement inputs, only limited studies have explored mmWave sensing modalities.The related work identifies RadarLLM and HoloLLM among recent mmWave-oriented attempts.
Sensing modalities with LLMs: Existing mmWave-LLM models struggle with real-world clutter or depend on modality-specific encoders, whereas textualized point clouds enable zero-shot LLM interpretation and benchmarked reasoning.Synthetic-radar models face real-world generalization problems, while raw-signal models require modality-specific encoders.

3. mmWave-QA Benchmark

mmWave-QA integrates three heterogeneous mmWave datasets into a standardized language-conditioned benchmark through textualization, taxonomy-based annotation, QA generation, and quality review. It covers 86 refined action categories, five question domains, and six scene conditions for evaluating human-motion understanding.

Dataset Integration: mmWave-QA integrates mmBody, MM-Fi, and mRI across multiple radar devices, environments, and subjects, refining 139 action labels into 86 distinct categories.Redundancies and ambiguities were removed during integration.
Data Collection and Textualization: Coordinate-based quantization standardizes heterogeneous point-cloud formats and converts dense radar data into a representation compatible with LLM token limits.RGB videos and skeleton videos provide visual references for annotators in clean, degraded, or occluded scenes.
Action Taxonomy and Annotation: The benchmark organizes human-motion understanding into five question domains: ActRec, TrajCheck, ActOrder, ActNum, and LimbFocus.These domains target recognition, trajectory checking, sequential reasoning, quantitative estimation, and localized motion analysis.
Question–Answer Generation: Annotators generate natural-language and multiple-choice QA pairs from labeled actions, while LLM paraphrasing increases linguistic diversity without changing semantic intent.Answer options are selected from the predefined action taxonomy.
Quality Review: Three annotators independently review generated QA pairs, removing any pair marked invalid by at least one reviewer to improve reliability and balance.The review addresses ambiguous semantics, inconsistent labels, and biased action distributions.
Benchmark Statistics: The benchmark spans six scene conditions—normal, furnished, rain, smoke, dark, and occlusion—although TrajCheck is omitted from the condition-distribution plot because of limited samples.ActRec and TrajCheck are the most frequent question types, while ActOrder, ActNum, and LimbFocus provide less frequent complementary coverage.

4. LLM-driven mmWave-QA

This section formalizes LLM inference on mmWave-QA and evaluates zero-shot, few-shot, chain-of-thought, and hybrid prompting strategies. The strategies vary the contextual inputs and reasoning process used to generate answers from questions and mmWave point-cloud sequences.

Inference formulation: LLM inference takes a natural-language question and a variable-density mmWave point-cloud sequence as inputs, together with an instruction prompt, to generate output tokens.Point density varies across sensing devices and environmental conditions.
Prompting strategies: The benchmark evaluates three prompting strategies: zero-shot, few-shot, and chain-of-thought inference.Each strategy conditions the model with different inputs, producing distinct reasoning and answer-generation behaviors.
Zero-shot: Zero-shot prompting generates an answer directly from the question–point cloud pair without additional examples or reasoning context.The generated answer is denoted Azero, and the corresponding model is Mzero.
Few-shot: Few-shot prompting supplies k question–point cloud–answer examples as contextual references, enabling inference without updating model parameters.The examples are represented as triplets (Qi, Pi, Ai)k_i=1.
Chain-of-Thought (CoT): Chain-of-thought prompting generates intermediate reasoning before the final answer, while a hybrid strategy combines CoT reasoning with few-shot examples.The rationale can explain inter-frame relationships within point clouds to support the final answer.

5. Experiments

Experiments show that prompting improves zero-shot LLM reasoning over heterogeneous mmWave radar data, with performance varying across actions, hardware, tasks, modalities, and frame counts. Radar also remains effective under visual degradation, while temporal and Doppler cues support particular reasoning tasks.

Prompting Strategies: GPT-5 achieved the best overall performance with few-shot CoT prompting, improving overall accuracy by 1.39 and 1.86 percentage points over GPT-4o and Gemini 2.5-Pro.Performance gains were consistent from zero-shot through few-shot, CoT, and few-shot CoT prompting across all evaluated models.
Prompting Strategies: Arm motions achieved the best action-category results, whereas leg and torso actions remained challenging because of subtle motion cues and frequent radar occlusions.Phoenix had the highest hardware accuracy, while IWR6843 and IWR1443 degraded due to sensor limitations.
Multi-aspect Reasoning: In zero-shot TrajCheck, GPT-4o and Gemini 2.5-flash achieved 92.0% and 87.2% accuracy for moving cases but only 4.0% and 3.2% for static cases.Few-shot examples increased static accuracy to 32.0% and 27.2%, respectively.
Multi-aspect Reasoning: Multi-limb scenarios produced lower accuracy than single- or limb-specific motions, likely because multipath interference and ghost reflections distort body-part separability.The findings motivate structural analysis of radar point clouds and selection of temporally salient frames for temporal reasoning.
Comparison with RGB Modality: RGB outperformed radar in normal and furnished environments, but ActNum consistently performed better with radar because motion counting relies on temporal Doppler cues.The comparison covered normal, furnished, rain, smoke, dark, and occlusion scenarios.
Frame-wise Performance Analysis: Performance peaked around 16 frames across most tasks as frame count increased from 6 to 64, balancing temporal coverage against information density.Too few frames limited temporal context, while excessive frames reduced performance.
Qualitative Analysis: Under darkness or occlusion, RGB-based reasoning often hallucinated or misinterpreted corrupted regions, whereas radar enabled stable Doppler-based motion reasoning.In ActRec, the LLM correctly inferred walking by tracking coordinate displacement and Doppler information.

6. Conclusion

The paper introduces mmWave-QA, a benchmark for evaluating diverse human actions through natural-language QA across heterogeneous mmWave sensing datasets. Experiments show that LLMs can interpret mmWave point clouds for QA and are more robust than RGB-conditioned models under visual degradation.

Benchmark: mmWave-QA is the first benchmark designed to assess diverse human actions through natural-language QA.It integrates multiple mmWave sensing datasets to capture variation across devices, environments, and scene conditions.
Benchmark: Hardware-aware preprocessing and global taxonomy alignment enable mmWave-QA to represent heterogeneity across sensing datasets.The benchmark accounts for differences in devices, environments, and scene conditions.
Findings: LLMs can effectively interpret mmWave point clouds and perform QA tasks, with greater robustness than RGB-conditioned models under low light or occlusion.This comparison uses the same underlying model conditioned on either mmWave point clouds or RGB inputs.