PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Fusion Transformer for Remote Sensing Image Classification

Swalpa Kumar Roy, Ankur Deria, Danfeng Hong, Behnood Rasti, Antonio Plaza, Jocelyn Chanussot

arXiv:2203.16952v2cs.CVcs.LGeess.IV

TL;DR

HSI classification needs effective use of complementary modalities while avoiding transformer complexity and weakly generalizing randomly initialized CLS tokens. MFT addresses this with modality-derived CLS tokens and mCrossPA, and the reported experiments show superior performance across the considered datasets and models.

  • Problem

    Concatenating HSI with complementary modalities increases transformer projection complexity, while randomly initialized CLS tokens may generalize poorly.

  • Method

    MFT generates a CLS token from corresponding multimodal data and fuses it with HSI patch tokens using multihead cross patch attention.

  • Results

    The proposed MFT performs better than all other tested models across all considered datasets.

  • Takeaways & Limitations

    MFT successfully fuses multimodal remote-sensing data within a transformer framework for land-cover classification.

Abstract

from arXiv · show

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a {distinctive representation} in a reduced and hierarchical feature space. Extensive experiments are carried out on {widely used benchmark} datasets {i.e.,} the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at \url{https://github.com/AnkurDeria/MFT}.}

I. INTRODUCTION AND CONTRIBUTIONS

Remote-sensing classification benefits from complementary multimodal data because single-sensor HSI can miss distinctions that LiDAR elevation information reveals. The proposed MFT fuses HSI with complementary modalities through mCrossPA and evaluates the approach on four public datasets.

  • Single-sensor HSI provides rich spectral and spatial information but may insufficiently identify objects of interest.
  • LiDAR adds elevation and intensity information that can distinguish land-cover objects with similar spectral signatures but different elevations.
  • MFT fuses HSI patch tokens with complementary multimodal data using a multihead cross patch attention mechanism.
  • The model derives supplementary class-token information from modalities such as LiDAR, MSI, SAR, and DSM alongside HSI patch tokens.
  • Experiments use four public hyperspectral datasets—Houston, Trento, MUUFL, and Augsburg—and compare HSI-only and multimodal configurations with existing methods.

B. Conventional Deep Learning Methods

Prior remote-sensing classification methods include conventional classifiers, deep CNN-based approaches, and transformer architectures. The reviewed methods address spectral-spatial representation and multimodal fusion, but transformers and CNNs retain complementary limitations.

  • B. Conventional Deep Learning Methods: Supervised deep-learning methods use ground-reference samples and include CNN1D, CNN2D, and CNN3D approaches for HSI and LiDAR classification.
  • B. Conventional Deep Learning Methods: CNN-based networks can inadequately capture subtle spectral discrepancies and middle- or long-term sequence dependencies.
  • C. Transformer-based Methods: Transformers use self-attention to model global correlations and capture spectral-signature information, but may inadequately characterize local semantic and spatial information.
  • C. Transformer-based Methods: Existing transformer variants explore spectral-spatial features, long-range dependencies, convolutional local features, and joint HSI–LiDAR or multisource classification.

B. Proposed Multimodal Fusion Transformer

The proposed MFT addresses the parameter-growth challenge of concatenating HSI and multimodal inputs by using a modality-derived CLS token and mCrossPA. CNN feature extraction and tokenization prepare HSI and LiDAR information for transformer fusion.

  • B. Proposed Multimodal Fusion Transformer: Concatenating HSI with LiDAR, SAR, or DSM increases projection complexity and may aggravate overfitting in transformer models.
  • B. Proposed Multimodal Fusion Transformer: MFT generates a CLS token from a corresponding multimodal patch and fuses it with HSI patch tokens through mCrossPA.
  • 1) HSI and LiDAR Feature Learning via CNNs:: The model uses CNNs to extract high-level abstract features and reduce HSI bands before transformer processing.
  • 1) HSI and LiDAR Feature Learning via CNNs:: Conv3D and HetConv2D extract robust HSI features, producing feature maps of shapes (8 × 11 × 11 × (B −8)) and (11 × 11 × 64).

2) HSI and LiDAR Tokenization:

The model tokenizes HSI patches and complementary LiDAR data into embeddings, using the LiDAR-derived classification token alongside HSI patch tokens. Positional embeddings and dropout are then applied before transformer encoding.

  • HSI Tokenization: HSI cubes of size (11 × 11) × 64 are flattened into (1 × 64) feature patch tokens, with n selected from 121 patches through tokenization.Tokenization uses two learnable weights, Wa and Wb, to extract key features.
  • LiDAR Tokenization: LiDAR patches are converted into a (1 × 64) CLS token using Conv2D, batch normalization, GELU activation, and either pixel or channel tokenization.The convolution preserves spatial height and width while adjusting the channel count for CLS construction.
  • Token Fusion: The LiDAR CLS token is concatenated with n HSI patch embeddings, producing n + 1 patches for transformer processing.The embedding size is 64, partly because it is a power of 2 that simplifies head-dimension calculation.
  • Embedding Preparation: Trainable position embeddings are added element-wise to the patch embeddings, followed by dropout with value 0.1.The position embeddings preserve positional and semantic textural information in the image patches.
  • Transformer Encoding: The transformer encoder combines residual multihead cross patch attention with a residual MLP block to process the fused tokens.The CLS token exchanges information with HSI patch tokens to learn an abstract representation of the HSI patch.

3) LiDAR and HSI Cross Patch Attention:

The mCrossPA module fuses a LiDAR-derived CLS token with HSI patch tokens through multihead cross attention. Its output is normalized, projected, passed through residual transformer blocks, and classified from the resulting CLS representation.

  • Cross-Modal Input: Generated HSI patch tokens are concatenated with a spatially corresponding LiDAR CLS token before position embedding, dropout, and cross patch attention.The two modalities are aligned by using LiDAR data from the same spatial region as the HSI patch.
  • Cross Patch Attention: CrossPA forms queries from the fused CLS token and keys and values from the combined embedding to generate the fused CLS output.A linear projection precedes dropout with value 0.1.
  • Multihead Attention: Using multiple heads strengthens connections between the various HSI patch tokens and the CLS token, yielding the mCrossPA mechanism.The proposed model sets the number of heads to 8.
  • Transformer Encoder: Each mCrossPA output passes through layer normalization and a residual shortcut, followed by an MLP block and another residual shortcut.The resulting representation becomes the input to the next transformer encoder block.
  • Classification: The CLS output from the final transformer encoder block is fed into the classifier head to produce the final classification results.The classifier uses the kth block output ycls = XHLk(1, :).

A. HSI Datasets

The evaluation uses four hyperspectral scenes and their associated multimodal sources to assess the proposed multimodal fusion transformer for land-cover classification. The datasets provide varied sensors, spatial resolutions, class structures, and sampling protocols.

  • Dataset Collection: Experiments cover the University of Houston, Trento, MUUFL Gulfport, and Augsburg scenes with associated LiDAR, MS, SAR, or DSM data.These four datasets are used to evaluate the proposed multimodal fusion transformer network.
  • University of Houston: The UH scene figure presents HSI, MS, and LiDAR imagery alongside disjoint training and test ground truths and class-specific sample counts.The HSI pseudo-color image uses bands 64, 43, and 22.
  • University of Houston: The University of Houston scene includes HSI, MS, and LiDAR images at 2.5-meter spatial resolution, with 15 land-cover and land-use classes.The images are 340 × 1905 pixels, and the HSI contains 144 bands while the MS image contains 8 spectral bands.
  • MUUFL Gulfport: MUUFL Gulfport contains a 325 × 220 HSI with 72 spectral bands, LiDAR elevation data, 11 urban land-cover classes, and 53,687 ground-truth pixels.After removing noisy initial and final bands, the HSI has 64 bands.
  • MUUFL Gulfport: The MUUFL figure shows HSI, LiDAR, and ground-truth data, with 5% of samples randomly selected for training and the remaining 95% used for testing.The HSI true-color image uses bands 40, 20, and 10.
  • Augsburg: Augsburg combines HSI, dual-Pol SAR, and DSM imagery, with all images down-sampled to a unified spatial resolution of 30 m GSD.The modalities come from HySpex, Sentinel-1, and DLR-3K systems.

B. Experimental Setup

The experiments compare MFT with conventional classifiers, CNNs, RNNs, and transformer models across HSI-only and multimodal settings. Evaluation uses controlled training configurations, repeated trials, standard metrics, and disjoint or varying train/test protocols.

  • Compared methods: MFT is compared with KNN, RF, SVM, CNN1D, CNN2D, CNN3D, RNN, ViT, and SpectralFormer.Comparisons use HSI alone and combinations with LiDAR, MSI, SAR, or DSM; other models concatenate modalities band-wise.
  • Implementation: The experimental setup runs on a 40-core server with 377 GB of RAM and a single Nvidia Tesla V100 GPU; MFT uses n = 4 HSI patch tokens.The reported configuration specifies Red Hat Enterprise Server 7.6 and 32510 MB of GPU memory.
  • Implementation: Training extracts 11 × 11 × B HSI patches and 11 × 11 × C multimodal patches, using batch sizes of 64 and 500 for training and testing.Deep models generally use Adam with learning rate 5e−4 and weight decay 5e−3; RNN uses learning rate 1e−3 without weight decay.
  • Implementation: Each experiment is repeated 3 times, with averages and standard deviations reported after 500 training epochs.A step scheduler uses step size = 50 and gamma = 0.9.
  • Metrics: Performance is assessed using overall accuracy, average accuracy, and statistical Kappa coefficients.OA measures the proportion of correctly classified test samples, while AA averages class-wise accuracy.
  • Evaluation settings: Experiments use disjoint spatially and spectrally separated samples, random training percentages, and model variations on the same disjoint datasets.The disjoint protocol keeps training and test intersections empty.

C. Performance Evaluation with Disjoint Train/Test Samples

Across disjoint train/test evaluations, MFT generally achieves the strongest classification performance across Houston, Trento, MUUFL, and Augsburg, although gains vary by dataset and modality. Fusion often improves results, while Augsburg is harder and includes an exception where ViT has higher AA.

  • Houston: MFT obtains the highest OA, AA, and κ on disjoint Houston evaluations across HSI-only, HSI-LiDAR, and HSI-MS settings.For HSI-only Houston, MFT reports mean OA 88.45%, AA 90.05%, and κ 87.46%.
  • Houston: 89.80% mean OA, 91.51% mean AA, and 88.93% mean κ are reported for Houston HSI-LiDAR fusion by MFT.These results are described as outperforming the remaining methods, with minimal standard deviations of 0.53%, 0.40%, and 0.59%, respectively.
  • Houston: 89.15% mean OA, 90.56% mean AA, and 88.22% mean κ are reported for Houston HSI-MS fusion by MFT.The proposed transformer fusion improves MFT while most other models perform worse than in the HSI-only setting.
  • Trento and MUUFL: 97.76±00.40% OA, 95.91±00.41% AA, and 97.00±00.53% κ lead transformer models on Trento HSI-only data; MUUFL reaches 94.18 ± 00.09% OA.MUUFL also reports AA =81.00 ± 00.55% and κ =92.30 ± 00.12%.
  • Trento and MUUFL: Adding LiDAR yields MFT gains of 0.56% OA, 0.07% AA, and 0.75% κ for Trento, and 0.26%, 0.48%, and 0.21% for MUUFL.The passage states that LiDAR and HSI improve all models except RF in Trento.
  • Augsburg: Augsburg is harder than the preceding datasets, yet MFT records the highest OA, AA, and κ except for the HS-SAR case.In that exception, ViT has 0.76% higher AA, while MFT has 4.39% higher OA and 6.20% higher κ; MFT reaches AA = 64.70 ± 00.44% in another case.

D. Visual Comparison

Multimodal inputs improve the visual quality of classification maps, with MFT producing more realistic and finer land-cover details through information exchange between HSI patches and complementary modalities.

  • The visual comparisons include conventional classifiers, CNN variants, RNN, ViT, SpectralFormer, and MFT across MUUFL and Augsburg multimodal datasets.
  • Multimodal feature extraction yields remarkable improvements in the visual quality of classification maps.
  • MFT produces more noticeable visual improvements than the compared methods in the classification maps.
  • Better information exchange between HSI patches and other modalities helps MFT generate more realistic and finer land-cover details.

E. Performance Over Varying Training Ratio

The study evaluates MFT and comparison models across randomly selected training ratios from 3% to 9% on four multimodal datasets. Performance improves with more training samples, while MFT particularly outperforms ViT on both Houston configurations.

  • Training-ratio protocol: Training samples are randomly selected at 3%, 5%, 7%, and 9%, with remaining samples reserved for testing.Other experimental parameters remain unchanged from previous sections.
  • Evaluation setup: Figures 14(a)–(l) compare AA, OA, and κ across four datasets and varying training-sample percentages.The datasets are Houston HSI+LiDAR, Trento HSI+LiDAR, MUUFL HSI+LiDAR, and Houston HSI+MS.
  • Training-ratio effects: 3% training samples produce inferior results to 9% training samples across all four datasets.The analysis motivates selecting an appropriate number of labeled samples to avoid under-fitting or over-fitting issues.
  • Comparative performance: Approximately 4% and 6% better performance than ViT is reported for MFT on Houston HSI+LiDAR and Houston HSI+MS, respectively.These improvements apply to all three accuracies across all evaluated training-sample percentages.
  • Comparative performance: MFT also performs better than ViT on the other datasets, although the improvement is smaller at larger training ratios.The supplied passage does not quantify these dataset-specific differences.

F. Model Stability Analysis

The stability analysis examines tokenization, patch size, and computational efficiency for MFT. An 11×11 patch window gives the most stable reported accuracies, while computational efficiency varies by dataset relative to ViT.

  • Ablation study: Tokenization dimension is identified as crucial to final classification performance and requiring fine-tuning alongside learning and hyper-parameters.The ablation compares pixel and channel tokenization across all four datasets and their combined multimodal data.
  • Impact of patch size: An 11×11 input-patch window achieves the best and most stable AA, OA, and κ across all experimentally evaluated datasets.The experiments use disjointed train/test samples.
  • Computational efficiency: MFT uses fewer parameters and FLOPs than ViT on the Trento HSI+LiDAR and MUUFL HSI+LiDAR datasets.The passage identifies ViT as the model with the second-best accuracies in these comparisons.
  • Overall model behavior: MFT successfully fuses multimodal data across the considered datasets and performs better than all tested models.The paper frames this result as evidence for transformer-based multimodal fusion in land-cover classification.
Loading 2203.16952v2…