CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications

Yubo Zhang; Yiyao Liu

Source-linked AI summary

CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications

Yubo Zhang, Yiyao Liu

arXiv:2608.15972v1Aug 16, 2026cs.CVcs.AI

TL;DR

Cross-scenario changes make vision–wireless representations difficult to transfer, especially without discrete beam-index labels during pretraining. CM-MAE uses soft contrastive alignment and masked cross-modal reconstruction, while differential-rate adaptation preserves pretrained features. It achieves a 4.61-point average transfer gain and 77.38% unseen-scenario accuracy under the reported protocol.

Problem
Vision–wireless representations must transfer across changing scenarios without relying on discrete beam-index labels during pretraining.
Method
CM-MAE combines beam-profile soft contrastive alignment with masked joint reconstruction and differential-rate fine-tuning for cross-scenario representation transfer.
Results
A 4.61-point average transfer gain was observed, while differential-rate adaptation raised unseen-scenario accuracy from 51.36% to 77.38%.
Takeaways & Limitations
CM-MAE supports cross-scenario representation transfer in the diagnostic setting, rather than proactive reduced-sweep beam prediction.
Takeaways & Limitations
Finer path-resolved alignment requires ray-traced correspondence labels unavailable in the reported real-measurement experiments.

Abstract

from arXiv · show

Synchronized camera and wireless measurements observe the same scene through different physical channels. The central difficulty is that a representation learned in one deployment can fail when viewpoint, traffic, illumination, and propagation geometry change. This paper presents CM-MAE, a self-supervised vision--wireless pretraining framework for cross-scenario representation transfer. The evaluated real-data model uses only RGB frames and the measured 64-beam received-power vector available in DeepSense 6G; it does not use ray-traced paths, calibrated depth, or beam-index labels during pretraining. Its central pretraining term is a \emph{soft contrastive alignment loss}. Instead of making the synchronized image--wireless pair the only positive pair, this loss builds a target distribution from similarities between measured beam-power profiles, so nonidentical samples with similar directional responses are not forced apart as false negatives. A masked joint decoder provides the complementary local objective by reconstructing hidden visual patches and wireless angular clusters under modality dropout. After pretraining, a differential-rate fine-tuning rule lets a new fusion head adapt quickly while the encoders move slowly. Under a sequence-disjoint DeepSense 6G protocol, adding the soft alignment loss improves a matched linear-probe transfer average from 24.88\% to 29.49\%. Mild fusion fine-tuning reaches 77.38\% Top-1 accuracy on unseen Scenarios 6--8, and optional transductive normalization adaptation reaches 78.69\%. Since the fusion setting uses the contemporaneous 64-beam power vector at inference, these results should be read as representation-transfer diagnostics, not as proactive beam-prediction or reduced-sweeping claims.

I. INTRODUCTION · II. BACKGROUND AND RELATED WORK · III. PROBLEM SETUP

CM-MAE targets transferable vision–wireless representations from synchronized RGB frames and measured 64-beam received-power vectors, withholding beam-index labels during pretraining. It combines beam-profile-guided soft contrastive alignment with masked conditional reconstruction and evaluates transfer across changing deployment scenarios.

I. INTRODUCTION: Changing camera placement, street layout, traffic, reflectors, blockers, and array response can disrupt the relationship between image pixels and selected wireless beams.
III. PROBLEM SETUP: Pretraining uses synchronized RGB images and 64-dimensional measured received-power vectors, while the strongest measured beam index is reserved for downstream classification.
I. INTRODUCTION: CM-MAE uses measured beam-profile similarity to form soft cross-modal retrieval targets rather than treating paired identity as the only positive relation.
I. INTRODUCTION: Masked conditional reconstruction complements global alignment through a joint decoder for visual patches and sparse one-dimensional angular clusters, with modality availability controlling masking and decoding.
I. INTRODUCTION: 4.61-point average transfer gain comes from soft alignment in matched ablations; differential-rate adaptation raises the unseen-scenario average from 51.36% to 77.38%, with optional transductive adaptation reaching 78.69%.
II. BACKGROUND AND RELATED WORK: Prior work spans contrastive and masked self-supervision, DeepSense 6G multimodal measurements, supervised vision-position beam prediction, and wireless foundation-style models.
II. BACKGROUND AND RELATED WORK: Cross-scenario evaluation tests transfer beyond source validation, while Tent and domain-adversarial training represent adaptation-only approaches using entropy minimization or gradient reversal.

A. Synchronized Image and Beam-Power Measurements

Each synchronized sample pairs an RGB frame with a measured received-power vector over a 64-beam codebook from a co-located camera and base station. The strongest beam index is reserved for downstream supervision, while the image tokenizer uses only a stable image-plane direction coordinate rather than depth or propagation-path labels.

Measurements: Each sample combines an RGB frame with received powers measured across DeepSense’s 64-beam sweep.The received-power vector contains one measured value for each beam and includes noise and measurement perturbations.
Measurements: The strongest measured beam index serves as the downstream class label, but CM-MAE pretraining does not consume that index as a label.Pretraining instead consumes the full measured power vector, which contains the maximizer used for the downstream label.
Image tokenization: The image tokenizer divides each frame into nonoverlapping S × S patches and derives a normalized camera ray from each patch center.The patch count is P = (H/S)(W/S), and the ray uses homogeneous patch-center coordinates with the camera intrinsic matrix.
Image tokenization: In the evaluated implementation, the camera ray is only a stable image-plane direction coordinate, not a depth estimate or propagation-path label.This coordinate comes from an approximate pinhole camera with a 90° field of view.

B. Learning Objective · IV. EVALUATED CM-MAE ARCHITECTURE · A. Tokenization

CM-MAE learns cross-modal representations through beam-profile soft alignment and masked conditional reconstruction, using CLS embeddings for global modality representations. Its evaluated tokenizer encodes camera direction and wireless angular structure while preserving the limitation that fusion inference uses contemporaneous wireless measurements.

B. Learning Objective: Vision and wireless encoders prepend learned CLS tokens, whose final-layer outputs serve as global modality representations.The CLS embeddings are denoted zᵛᵢ ∈ R^d.
B. Learning Objective: CM-MAE pretraining combines beam-profile soft alignment with masked conditional reconstruction of hidden visual patches and wireless cluster features.Soft alignment uses normalized beam-power-profile similarity to shape cross-modal retrieval probabilities rather than treating only synchronized pairs as positives.
B. Learning Objective: Fusion inference supplies both the image and contemporaneous 64-beam power vector, evaluating cross-scenario multimodal representation transfer.This protocol does not demonstrate proactive prediction before a sweep or reduced beam-training overhead.
A. Tokenization: Vision tokens combine convolutional patch embeddings, learned patch-position embeddings, and approximate camera-ray coordinates.The camera-ray coordinate provides directional consistency without unsupported depth or propagation-path claims.
A. Tokenization: Wireless received power is converted to dB and normalized per sample, producing an angular-shape vector in [0, 1]^N_b while embedding maximum power as a scale token.The normalization removes absolute power, so M_i is separately embedded.
A. Tokenization: The evaluated tokenizer converts each dense 64-beam profile into a short sequence by retaining up to Q_max = 10 strongest local peaks above η = 0.15.If no local peak passes the threshold, the global maximum is used as a fallback.
A. Tokenization: Wireless cluster tokens encode angular location, local lobe shape, relative energy, peak quality, Fourier angular structure, and absolute scale without inferring physical propagation paths.Padded cluster slots are excluded through the attention mask.

B. Dual Transformer Encoders and Joint Decoder

CM-MAE uses separate pre-LayerNorm Transformer encoders for visual and wireless sequences, followed by a joint decoder during self-supervised pretraining. The decoder reconstructs modality-specific outputs and is removed from downstream classification, which uses unmasked encoders and a separate fusion head.

The visual and wireless streams use separate Vision Transformer encoders rather than shared weights, with each encoder implemented as a pre-LayerNorm Transformer stack.
The evaluated Base configuration uses width d = 768, 12 encoder layers, 12 attention heads, and MLP ratio 4 in each encoder.The model has roughly 232M parameters including the decoder and task head.
During pretraining, the joint decoder receives the concatenated encoded sequence with modality-type embeddings.
The decoder has eight Transformer layers, width 768, and 12 attention heads, with separate heads predicting denormalized RGB patch vectors and six-dimensional local wireless features.
The decoder is used only for self-supervised pretraining; downstream classification uses the unmasked encoders and a separate fusion head.

C. Soft Contrastive Alignment Loss

CM-MAE replaces hard one-hot image–wireless instance matching with a symmetric soft alignment objective whose targets encode measured beam-profile affinities. The loss therefore matches cross-modal retrieval neighborhoods while avoiding false negatives among samples with similar wireless responses.

C. Soft Contrastive Alignment Loss: Hard instance matching treats every nonpaired sample as a negative, which is inappropriate when different samples have similar measured beam-power shapes.CM-MAE uses the wireless response to determine how strongly each sample should associate with other batch samples.
C. Soft Contrastive Alignment Loss: CM-MAE defines soft targets from normalized measured power profiles, with the evaluated implementation setting the diagonal to zero for non-self neighborhood learning.The synchronized pair remains coupled through joint reconstruction; σh = 0.5, with smaller values concentrating probability on nearest non-self profiles.
C. Soft Contrastive Alignment Loss: The symmetric soft alignment loss matches learned cross-modal retrieval distributions to wireless-response neighborhoods rather than using a many-positive binary contrastive objective.A nonpaired sample with large qij is pulled closer until its learned probability matches measured beam-profile affinity, whereas qij ≈0 remains negative.
C. Soft Contrastive Alignment Loss: “Physics-grounded” refers only to computing targets from measured propagation responses, not to path recovery or universal electromagnetic invariance.The method’s physical guidance comes from measured response structure rather than explicit claims of recovering propagation paths.

D. Masking, Modality Dropout, and Reconstruction

CM-MAE masks visual and wireless tokens according to their information density while preserving coordinates, then uses joint reconstruction and modality dropout to make pretraining conditional on available streams. The alignment term is gated off when either CLS embedding is absent, but reconstruction continues for present branches.

Masking: Vision uses a 0.75 mask ratio with contiguous four-patch blocks, while wireless uses 0.5 masking over valid angular-cluster tokens.The different ratios reflect dense, spatially redundant images versus sparse wireless clusters.
Masking: Masked tokens retain coordinate embeddings while their content is replaced by learned mask tokens, preserving the locations of hidden patches and angular clusters.Position information distinguishes missing wireless clusters at different beam angles.
Reconstruction: The joint decoder reconstructs hidden visual patches and wireless clusters from both streams when available, rather than relying on isolated unimodal branches.This is a pretraining design choice, not a guarantee that reconstruction alone ensures transfer.
Modality Dropout: Modality dropout independently drops vision or wireless with probability 0.1, preventing the decoder from assuming both modalities are always present.The implemented loss supports missing-branch conditions during pretraining.
Modality Dropout: When either CLS embedding is absent, the product gate removes alignment loss while reconstruction remains active for whichever branch is present.This keeps reconstruction usable under modality dropout.

V. LEARNING AND ADAPTATION PROCEDURE · A. Stage 1: Self-Supervised Pretraining · B. Stage 2: Source-Supervised Adaptation

CM-MAE pretrains on paired unlabeled camera–wireless samples using masked reconstruction and conditional soft contrastive alignment, without beam-index labels. Source-supervised adaptation then uses an unmasked fusion head with two-timescale optimization to limit pretrained encoder drift.

A. Stage 1: Self-Supervised Pretraining: Stage 1: Self-Supervised Pretraining: Pretraining optimizes two encoders and a joint decoder on paired but unlabeled source-scenario samples with visual, scale, and wireless angular-cluster tokens.Independent masks and modality-dropout indicators are sampled for each mini-batch.
A. Stage 1: Self-Supervised Pretraining: Stage 1: Self-Supervised Pretraining: Beam-index label b⋆i is excluded from pretraining; soft contrastive alignment requires both modality branches, while reconstruction uses every available branch.Checkpoint selection uses the source validation scenario after downstream adaptation, not target-scenario labels.
B. Stage 2: Source-Supervised Adaptation: Stage 2: Source-Supervised Adaptation: After pretraining, unmasked encoders produce features, and fusion adds a projection of the full normalized power vector to the wireless CLS feature.No reconstruction mask is applied during adaptation.
B. Stage 2: Source-Supervised Adaptation: Stage 2: Source-Supervised Adaptation: A bidirectional residual attention module exchanges visual and wireless global features before an MLP predicts one of Nb = 64 beam indices.The attention module is two-layer and operates on the global features.
B. Stage 2: Source-Supervised Adaptation: Stage 2: Source-Supervised Adaptation: Uniform learning rates caused large representation drift in cross-scenario runs, motivating separate optimizer groups for pretrained encoders and new fusion-classifier parameters.The update rule distinguishes pretrained encoder parameters θe from newly initialized fusion and classifier parameters θh.
B. Stage 2: Source-Supervised Adaptation: Stage 2: Source-Supervised Adaptation: Encoder and head learning rates are ηe = 3 × 10−6 and ηh = 10−3, respectively.AdamW additionally uses cosine decay, five-epoch warmup, weight decay 0.1, label smoothing 0.05, head dropout 0.15, gradient clipping at 1, and 80 epochs.
B. Stage 2: Source-Supervised Adaptation: Stage 2: Source-Supervised Adaptation: The peak head rate is about 333 times the encoder rate, enabling rapid task-specific head adaptation while limiting movement from the pretrained encoder.This is the intended two-timescale behavior of the final adaptation rule.

C. Stage 3: Optional Test-Time Adaptation … B. Main Results

CM-MAE is evaluated for cross-scenario transfer on sequence-disjoint DeepSense 6G data, where fusion with contemporaneous vision and wireless measurements generalizes strongly to unseen scenarios. Optional transductive normalization adaptation improves results, while geometry-privileged extensions remain outside the evaluated method.

C. Stage 3: Optional Test-Time Adaptation: Test-time adaptation updates only affine normalization parameters for 30 Adam steps at learning rate 5 × 10−4 on unlabeled target-scenario batches.Adaptation is performed separately for each target scenario, with source weights restored between scenarios.
D. Out-of-Scope Privileged Geometry: CM-MAE-W is reserved for a geometry-privileged extension and is not evaluated because DeepSense experiments lack ray-traced path labels, calibrated depth, and patch–path associations.The proposed extension would require complex wideband CFR and 3-D angle–delay clusters.
A. Setup: DeepSense 6G provides synchronized 224×224 RGB images and 64-beam power vectors across eight geographic scenarios at 60 GHz.The dataset contains real-world vehicle-to-infrastructure measurements.
VI. EXPERIMENTS: The protocol splits each scenario 70/15/15 by seq_index, pretrains on unlabeled Sc1–5, fine-tunes on labeled Sc1–4, validates on Sc5, and tests on held-out Sc6–8.The sequence-based split avoids leakage from consecutive frames of the same vehicle.
B. Main Results: Reported ablations isolate masking, soft-contrastive alignment, reconstruction, frozen linear probing, full fine-tuning, single-modality fine-tuning, and mild fine-tuning with or without TTA.Table II compares frozen linear probing with mild fine-tuning for the two strongest fusion pretraining variants.
B. Main Results: 77.4% Top-1 accuracy on Sc6–8 is achieved by CM-MAE with mild fine-tuning, outperforming the ASU baseline by +24.9 pp.The evaluation reports cross-scenario Top-1 accuracy on unseen test scenarios.
C. Stage 3: Optional Test-Time Adaptation: 78.7% Top-1 accuracy on Sc6–8 is reached after test-time adaptation, adding another +1.3 pp over mild fine-tuning.Adaptation is reported separately from the source-only mild-fine-tuning result.
B. Main Results: Sc6 contributes +22.5 pp and Sc7 +31.6 pp, identifying the hardest scenarios as low-light residential and dense urban blockage, respectively.These are the largest reported gains over the ASU baseline.

C. Ablation Studies · D. Comparison with Published Methods

The ablations show that soft physics-grounded alignment, masked completion, and restrained adaptation are central to transfer, while masking choices materially affect representation quality. Published-method comparisons require caution because most prior results are in-distribution, making ASU ResNet the only directly matched cross-scenario reference.

C. Ablation Studies: Soft targets outperform hard targets across masking settings, confirming that physics-grounded partial credit improves representation learning.CLS-SoftCtr exceeds CLS-InfoNCE by +2.6 pp, while CM-MAE-U exceeds MAE+InfoNCE by +5.9 pp.
C. Ablation Studies: 38.1% linear-probe accuracy favors CLS-SoftCtr over the full model’s 31.7%, but mild fine-tuning reverses the ranking.CM-MAE-U reaches 45.0% on Sc6–8 versus 37.3% for CLS-SoftCtr, a +7.7 pp gain.
C. Ablation Studies: 80.1% versus 61.3% on Sc8 shows that masked completion’s advantage concentrates in the hardest unseen scenario.Masked completion supplies local spatial features that mild fine-tuning exploits for new-scenario adaptation.
C. Ablation Studies: 77.4% accuracy from mild fine-tuning exceeds full fine-tuning’s 51.4% by +26.0 pp, showing that adaptation strategy is as important as pretraining.Mild fine-tuning preserves cross-modal features, whereas full fine-tuning fits source scenarios but forgets pretrained knowledge.
C. Ablation Studies: 31.7% with vision masking ratio rv=0.75 beats rv=0.5 at 28.8% and rv=0.9 at 27.4%, producing an inverted-U pattern.CSI masking also shows a similar pattern, with rc=0.75 hurting most by −6.3 pp.
C. Ablation Studies: Block masking beats uniform random masking by +2.5 pp, while disabling adaptation loss with λda=0 consistently gives the best result.Continuing SSL pretraining on all eight scenarios with a freshly reset optimizer degraded Sc6–8 accuracy to ∼49%.
D. Comparison with Published Methods: Table VI is a positioning summary rather than a protocol-matched leaderboard because most prior methods report in-distribution results.The ASU ResNet cross-scenario reference is the only directly matched comparison identified in the table.

VII. ANALYSIS AND DISCUSSION … VIII. CONCLUSION

CM-MAE’s cross-modal alignment supports transfer across difficult scenarios, while single-modality results expose the limits of scenario-specific wireless information and weak vision-only features. The analysis also identifies limitations in correspondence granularity, temporal modeling, continued pretraining, evaluation scope, and deployment cost.

A. Per-Scenario Analysis: 92.5% on Sc8 makes it the easiest scenario, while Sc6 and Sc7 improve over the ASU baseline by +22.5 and +31.6 pp, respectively.Sc8 has 652 test samples and structured propagation; Sc6 and Sc7 have 254 and 159 samples, respectively.
A. Per-Scenario Analysis: 81.4% Sc5 accuracy from CSI-only mild-FT contrasts with poor Sc6/7 transfer at 18.9% / 22.6%, while vision-only reaches only 43.0% on Sc5.The asymmetry indicates strong in-domain CSI performance but weak cross-scenario transfer, alongside generally weak vision-only results.
A. Per-Scenario Analysis: Raw power vectors contain strong scenario-specific information, whereas transferable vision features require CSI alignment; their combination enables generalization.The passage identifies cross-modal combination, rather than either single stream alone, as the source of generalization.
B. Effect of the Soft Contrastive Targets: Beam-power correlations provide physically grounded soft targets because similar profiles indicate shared propagation characteristics, while σh=0.5 smooths noisy real measurements.This target construction differs from image-augmentation positives, which are not relevant for CSI.
C. Limitations and Future Work •: Sample-level CLS alignment is coarser than patch-to-beam-cluster matching, whose CM-MAE-W formulation requires simulator-only AoA/AoD and path-power labels.Synthetic-data evaluation and transfer to real measurements remain ongoing.
C. Limitations and Future Work •: Single-frame prediction omits short image–CSI sequences that could improve robustness to transient occlusions.Temporal modeling is proposed as a future extension, including sequences used in the DeepSense challenge task.
C. Limitations and Future Work •: All-scenario continued pretraining produced a negative result, motivating further study of optimizer-state handling and learning-rate scheduling; hidden-test evaluation could add external comparison.The passage also identifies the hidden DeepSense challenge test set as an unbiased evaluation opportunity.
VIII. CONCLUSION: ∼232 M parameters make the dual-stream ViT-Base architecture computationally costly, motivating distillation into a smaller deployment model.The conclusion reports CM-MAE as a physics-guided cross-modal masked autoencoder for 6G beam-classification transfer, reaching 77.4% Top-1 on Sc6–8.