Millimeter Wave Drones with Cameras: Computer Vision Aided Wireless Beam Prediction

I. INTRODUCTION

• Large antenna arrays and narrow directive beams create substantial training overhead, making frequent accurate beam selection difficult for highly mobile drones.Classical methods typically save only one order of magnitude in training overhead, while prior sensing-based work often considers users with more restricted, two-dimensional motion.Source: §I · p. 1
• Drone mobility and multiple possible orientations challenge beam prediction, while position-only approaches may be affected by GPS errors in real-world scenarios.Source: §I · p. 1
• The paper formulates vision-aided beam prediction for mmWave/THz drone communication using practical visual and communication models.Source: §I · p. 1
• A deep-learning solution uses visual data captured by cameras installed on drones to predict their beams.Source: §I · p. 1
• The ViWi-Drone dataset, based on the ViWi framework, contains coexisting multimodal visual and wireless data for drone beam prediction.Source: §I · p. 1
• More than 90% top-1 beam prediction accuracy is achieved on the ViWi-Drone dataset.The result highlights the potential of visual data to reduce beam-training overhead.Source: §I · p. 1

II. SYSTEM MODELS

• The base station has an M-element uniform linear array, while the drone has a single-antenna receiver and three RGB cameras; transmission uses OFDM with K subcarriers and cyclic prefix length D.Source: §II · p. 2
• The downlink channel h_k[t] is an M-dimensional vector for subcarrier k and time t, and the codebook contains Q candidate beamforming vectors.Source: §II · p. 2
• The received signal model includes the channel, selected beamforming vector, transmitted symbol, and additive complex-Gaussian noise.The transmitted symbol has average power P, and the noise variance is σ^2.Source: §II · p. 2
• The base station uses a predefined codebook, and the beamforming vector at each time step is selected to maximize average receive SNR.Source: §II · p. 2

III. PROBLEM FORMULATION

• Exhaustive codebook search incurs large beam-training overhead, while acquiring the channel information needed to compute optimal beams is generally hard.These constraints make beam prediction challenging for highly mobile drones.Source: §III · p. 2
• The prediction function maps a drone-captured RGB image X[t] to an estimate of the optimal beam index.The target is the beam that maximizes receive SNR.Source: §III · p. 2
• The model learns from labeled image–beam pairs to maximize correct predictions across the dataset, assuming samples are independently and identically distributed.The dataset is denoted D, and the learned model parameters are Θ.Source: §III · p. 2

IV. PROPOSED CAMERA AIDED SOLUTION

• The solution uses visual data captured by mmWave drones to predict downlink optimal beam indices.The section describes a key idea followed by an in-depth overview of the solution.Source: §IV · pp. 2–3
• Fig. 2 shows the different street regions captured by three cameras and the basestations visible from cameras 2 and 3.BS1 is seen from camera 2 and BS2 from camera 3.Source: Figure 2 · p. 3
• Fig. 3 presents a top view of the ViWi-Drone scenario, modeled on a busy downtown street, and highlights the drone trajectory.The scenario includes diverse objects such as cars, buses, and trucks.Source: Figure 3 · p. 3

A. Key Idea

• MmWave path loss makes line-of-sight communication preferable, paralleling cameras’ capture of visible or line-of-sight objects.This motivates using visual data for mmWave drone beam prediction.Source: A. Key Idea · p. 3
• A predefined beam codebook divides space into beam-associated sectors, allowing a beam index to be assigned from the user’s location in the visual scene.The task is framed as classification.Source: A. Key Idea · p. 3
• The approach computes optimal beam indices from drone-camera visual data instead of performing conventional beam training.The visual scene can support detecting objects and extracting the user’s relative position.Source: A. Key Idea · p. 3

B. Machine Learning Model

• The model must provide accurate predictions with low inference latency, so the solution uses an ImageNet2012-pretrained ResNet for beam prediction.Residual skip connections are described as supporting accuracy with fewer parameters, and ResNet is noted to have fewer FLOPs than VGG-Net.Source: B. Machine Learning Model · p. 3
• The comparison considers ResNet-18 and ResNet-50, representing smaller and larger residual-network architectures.Both pretrained models are fine-tuned on the image–beam-pair dataset.Source: B. Machine Learning Model · p. 3
• The pretrained models’ final classification layers are replaced with fully connected layers having Q output neurons for beam prediction.The models are then fine-tuned on labeled image–beam pairs.Source: B. Machine Learning Model · p. 3

C. Pruning Filters and Feature Maps

• Low-ranking filters are pruned by their L1-norm weight importance, then the network is fine-tuned to recover accuracy while reducing inference computation.The method removes a percentage of low-ranked filters and aims to minimize accuracy loss.Source: C. Pruning Filters and Feature Maps · pp. 3–4

V. VIWI-DRONE DATASET

• ViWi-Drone provides RGB image–beam-index pairs from a simulated downtown street, with a drone at 50 meters and three cameras capturing different directions.The scenario is built using the ViWi framework and includes two basestations spaced 100 meters apart, each with a 128-antenna array.Source: §V · p. 4
• The dataset contains 6,735 samples across 17 trajectories, with flight height fixed at 50 meters and paths running linearly above the street.The trajectories vary along the x and y axes; the drone travels in either the positive or negative x direction.Source: §V · p. 4
• Fig. 4 plots top-k beam prediction accuracy for ResNet-18 and ResNet-50.Source: Figure 4 · p. 4
• Samples are organized into BS1, BS2, and combined scenarios, each using a 70% training and 30% validation split.Source: §V · p. 4

• Both ImageNet2012-pretrained ResNet models are modified to output 128 neurons and trained with cross-entropy loss and Adam.Experiments use a single NVIDIA RTX TITAN GPU with PyTorch.Source: §VI · p. 4
• Top-k accuracy measures the percentage of test samples whose ground-truth beam appears among the k highest-ranked predictions.The evaluation also includes a study of post-training inference latency.Source: §VI · pp. 4–5
• Fig. 5 presents combined-dataset top-1 confusion matrices for both ResNets and beam-range accuracies, while Fig. 6 plots top-1 and top-3 accuracy against dataset size.Source: Figure 5 · p. 5, Figure 6 · p. 5

A. Beam Prediction Performance

• Across all three scenarios, top-1 accuracy exceeds 90% and top-3 accuracy is close to 100%; ResNet-50 improves performance by 1–2% over ResNet-18.ResNet-50 has 23M parameters, almost twice ResNet-18’s 11M, creating a performance–computation trade-off.Source: A. Beam Prediction Performance · p. 5
• For beam indices 30–50, accuracy is 89.85% for ResNet-18 and 92.08% for ResNet-50; ResNet-50 leads by almost 6% at indices 50–70.Accuracy is low for indices 70–90, primarily because the training dataset has fewer samples in that range.Source: A. Beam Prediction Performance · pp. 5–6
• More than 80% top-1 accuracy is achieved using 50% of the training data, around 2,350 of 4,715 samples.The full dataset has around 6,735 data points, divided into 70% training and 30% validation.Source: A. Beam Prediction Performance · p. 6

B. Inference Latency

• ≈95% of filters can be pruned without a significant accuracy drop, resulting in ≈30% improvement in inference latency.The analysis used a proposed ResNet-18 architecture on an 8GB NVIDIA RTX 2070 GPU.Source: B. Inference Latency · p. 6

VIII. CONCLUSION

• The approach uses visual data to predict optimal beam indices in mmWave/THz drone communication systems.The solution is evaluated on the ViWi-Drone dataset, which consists of diverse wireless and visual data.Source: §VIII · p. 6
• The vision-aided solution achieves ≈91% top-1 and 100% top-3 beam prediction accuracy.The conclusion also reports near-real-time prediction latency of 4.36 ms without significant reduction in prediction accuracy after network pruning.Source: §VIII · p. 6
• The experimental results indicate gains from leveraging RGB images to reduce beam training overhead in mmWave/THz drones.Source: §VIII · p. 6