PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification

Yuxiang Zhang, Mengmeng Zhang, Wei Li, Shuai Wang, Ran Tao

arXiv:2209.02700v1cs.CV

TL;DR

HSI domain generalization has largely neglected linguistic prior knowledge that could support cross-scene classification. LDGnet trains on source-domain data, aligns visual features with coarse- and fine-grained text in a shared semantic space, and reports superior performance across three datasets compared with state-of-the-art methods.

  • Problem

    Most HSI domain generalization methods learn visual representations without mining linguistic prior knowledge about land-cover classes.

  • Method

    LDGnet uses dual image and text encoders, coarse- and fine-grained text representations, and supervised contrastive visual-linguistic alignment in a cross-domain shared semantic space.

  • Results

    LDGnet demonstrates superiority over state-of-the-art techniques in experiments on three datasets.

  • Takeaways & Limitations

    Linguistic modality provides supervised semantic signals that assist visual representation learning and cross-domain invariant representation learning.

Abstract

from arXiv · show

Text information including extensive prior knowledge about land cover classes has been ignored in hyperspectral image classification (HSI) tasks. It is necessary to explore the effectiveness of linguistic mode in assisting HSI classification. In addition, the large-scale pre-training image-text foundation models have demonstrated great performance in a variety of downstream applications, including zero-shot transfer. However, most domain generalization methods have never addressed mining linguistic modal knowledge to improve the generalization performance of model. To compensate for the inadequacies listed above, a Language-aware Domain Generalization Network (LDGnet) is proposed to learn cross-domain invariant representation from cross-domain shared prior knowledge. The proposed method only trains on the source domain (SD) and then transfers the model to the target domain (TD). The dual-stream architecture including image encoder and text encoder is used to extract visual and linguistic features, in which coarse-grained and fine-grained text representations are designed to extract two levels of linguistic features. Furthermore, linguistic features are used as cross-domain shared semantic space, and visual-linguistic alignment is completed by supervised contrastive learning in semantic space. Extensive experiments on three datasets demonstrate the superiority of the proposed method when compared with state-of-the-art techniques.

I. INTRODUCTION

Cross-scene HSI classification suffers from domain shifts that limit fixed-scene CNN methods, while existing DG approaches largely omit linguistic prior knowledge. LDGnet addresses this gap by aligning visual features with coarse- and fine-grained language representations in a shared semantic space.

  • HSI acquisition factors such as sensor nonlinearities, seasons, and weather alter spectral reflectance between source and target scenes.
  • CNN-based methods trained for fixed scenes exhibit high generalization error and poor interpretation in cross-scene classification.
  • Existing DG methods mainly learn visual domain-invariant representations and do not use language knowledge for visual representation learning.
  • LDGnet treats language knowledge as shared between source and target domains and uses semantic space to unify visual and linguistic modalities.
  • The network combines image and text encoders with visual-linguistic alignment, using supervised contrastive learning to project visual features into the semantic space.
  • Coarse-grained and fine-grained text representations enrich supervised signals and promote domain-invariant representation learning.

II. RELATED WORK

Domain adaptation accesses target-domain data during training, whereas domain generalization learns from source domains without accessing the target domain. Prior DG work reduces domain shifts through invariant representations or data manipulation, while image-text foundation models learn joint cross-modal representations through large-scale pretraining.

  • Domain generalization trains on labeled source domains without accessing target-domain data, unlike domain adaptation.
  • Existing DG methods are grouped into domain-invariant representation learning and data manipulation.
  • Representation-based DG reduces shifts between source domains through explicit alignment using measures such as MMD, second-order correlation, or Wasserstein distance.
  • Data-manipulation DG augments or generates out-of-domain samples, often with VAE or GAN models, before transfer to the target domain.
  • Image-text foundation models learn joint visual-linguistic representations from large-scale image-text pretraining and support transfer across downstream tasks.

III. PROPOSED SINGLE-SOURCE DOMAIN EXPANSION NETWORK

LDGnet combines image and text encoders with visual-linguistic alignment to train on source-domain HSI patches and transfer image-only prediction to target domains.

  • Architecture: LDGnet processes source-domain image patches through an image encoder, classification head, projection head, and text encoder for multimodal learning.The image encoder extracts visual features, while coarse- and fine-grained text representations form the linguistic component.
  • Visual-linguistic alignment: Supervised contrastive learning aligns visual and linguistic features by class in a shared semantic space.This alignment is intended to reduce the gap between modalities and support target-domain prediction using the image encoder and classifier.
  • Image encoder: The image encoder uses deep residual 3D CNN modules to extract spatial-spectral features from hyperspectral data.Its architecture contains two 3D residual Block-MaxPool3d modules and one Conv3d module.
  • Training objectives: Classification uses cross-entropy on visual predictions, while projected visual features are used for visual-linguistic alignment.The classification head receives extracted features, and the projection head produces features for alignment with linguistic representations.

B. Text encoder

The text encoder converts class-specific coarse- and fine-grained descriptions into linguistic features, which are aligned with visual features through supervised contrastive learning.

  • Text encoder: The text encoder is a 33M-parameter, 3-layer transformer with 512-wide representations, 8 attention heads, and a 76-token maximum sequence length.It uses lowercased byte pair encoding with a vocabulary of 49,152 and projects normalized linguistic features into semantic space.
  • Text representations: Coarse-grained descriptions use class-name templates, while fine-grained descriptions encode manually specified color, shape, distribution, and adjacency information.Examples include pale-green grass, trees beside roads, and elongated road shapes.
  • Visual-linguistic alignment: Supervised contrastive learning aligns image-text pairs by class using positive and negative sample sets.The alignment includes image-to-text and text-to-image losses, with a temperature parameter controlling softmax logits.
  • Optimization and inference: LDGnet jointly trains image and text encoders, combines classification and alignment losses, and uses only the image encoder and classifier during target-domain testing.The total loss balances alignment with λ and controls coarse- versus fine-grained contributions with α.

D. Generalization Performance of LDGnet

LDGnet improves class separability by using language as an additional supervised signal and embedding visual features in a shared semantic space.

  • Target-domain generalization: Visual features from Houston 2018 and Pavia Center target domains become more class-separable after embedding in LDGnet’s semantic space.The reported visualizations show improved aggregation of same-class features, including the first class in Houston 2018 and the third class in Pavia Center.

IV. EXPERIMENTAL RESULTS AND DISCUSSION

Experiments evaluate LDGnet on three cross-scene HSI datasets spanning Houston, Pavia, and GID source-target settings. The datasets differ in scenes, sensors, spatial resolutions, and class configurations.

  • Experimental datasets: Three cross-scene experiments use Houston, Pavia, and GID datasets to validate LDGnet.The compared methods include domain adaptation and domain generalization algorithms.
  • Houston dataset: Houston 2013 and Houston 2018 contain seven consistent classes but differ in sensors, spectral bands, spatial resolutions, and acquisition years.Houston 2013 has 144 bands and 2.5m resolution, whereas Houston 2018 has 48 bands and 1m resolution.
  • Pavia dataset: The Pavia dataset pairs University of Pavia and Pavia Center scenes with seven shared classes and aligned spectral-band counts after removing one University of Pavia band.Pavia Center has 102 bands, while University of Pavia is adjusted to the same number.
  • GID dataset: GID uses Nanchang as the source domain and Wuhan as the target domain, with matching spatial and spectral resolutions and five shared classes.The scenes were acquired at different times and locations in China.

B. Experimental Setting

The experimental setting uses LDGnet’s image-text implementation with coarse- and fine-grained linguistic descriptions. Fine-grained descriptions are manually specified for the evaluated datasets.

  • Implementation: LDGnet uses a 13×13 input patch and optimizes its image and text encoders with Adam.The default weight-decay value is 1e-4, and the text encoder is initialized from CLIP-pretrained ViT-B-32.pt.
  • Text descriptions: A coarse-grained description is generated for each class from the template “A hyperspectral image of <class name>”.The template is used to initialize class-level text descriptions.
  • Dataset setting: Table IV specifies the source and target sample counts for the GID dataset.The supplied setting identifies the table as the dataset’s source-target sample summary.
  • Text descriptions: Fine-grained text descriptions are artificially defined for the three datasets.The descriptions are listed in the dataset-specific Tables V–VII.

C. Parameter Tuning

Parameter tuning evaluates learning rate, regularization, and modality weight choices across the experimental datasets. The reported settings select different optimal regularization and modality-weight values by dataset.

  • Hyperparameter search: The base learning rate η, regularization parameter λ, and weight α are tuned over predefined candidate sets.The candidate ranges are reported for each adjustable hyperparameter.
  • Learning rate: 1e-2 is the ideal base learning rate across the three datasets.Table VIII reports classification results for the tested learning rates.
  • Regularization: The optimal λ is 1e+0 for Houston and GID, but 1e-2 for Pavia.These values are selected from the overall classification accuracies in Table IX.
  • Modality weight: The optimal α is 0.3 for Houston and Pavia, and 0.1 for GID.Table X reports overall classification accuracy for the tested modality weights.

D. Ablation Study

Ablation experiments remove the text encoder, projection head, or one text-granularity branch to assess LDGnet components. The reported comparisons favor linguistic supervision and the fusion of coarse- and fine-grained features.

  • Ablation variants: The ablation variants delete the text encoder and projection head, fine-grained text, or coarse-grained text.They are named LDGnet (cls), LDGnet (coarse), and LDGnet (fine), respectively.
  • Linguistic modality: Adding a text encoder with either coarse- or fine-grained descriptions improves performance by around 6% over LDGnet (cls).The comparison is made by adding the transformer text encoder to the classification-only variant.
  • Text-granularity fusion: LDGnet performs 1%∼4% better than either the coarse-only or fine-only variant.The result indicates that combining both linguistic granularities outperforms using either one alone.
  • Fine-grained text: Fine-grained text descriptions encode color, shape, distribution, and adjacency relationships for land-cover classes.The fine-grained text analysis evaluates whether these descriptions are relevant to the represented classes.
  • Fine-grained text: Irrelevant fine-grained descriptions reduce accuracy by 3% relative to 80.25% with artificially defined descriptions.Relevant Internet descriptions are 1% lower than the artificially defined descriptions, and both relevant sources outperform unrelated text.

F. Performance on Cross-Scene HSI Classification

LDGnet achieves stronger cross-scene target-domain classification than the compared DA and DG methods while using only source-domain labels for training. Its visual-linguistic alignment yields less noisy classification maps, but the text encoder makes computation relatively costly.

  • Classification maps: The GID-wh target-scene map reports 77.08% for LDGnet, exceeding LDSDG at 76.48% and DSAN at 73.69%.The figure also reports DAAN at 68.06%, MRAN at 67.49%, HTCNN at 57.20%, PDEN at 67.71%, and SagNet at 61.64%.
  • Quantitative comparison: LDGnet outperforms DSAN on all target domains by 4% to 6%.DSAN accesses target-domain data during training, whereas LDGnet uses self-supervised contrastive learning to align visual and linguistic features in a shared semantic space.
  • Quantitative comparison: LDGnet outperforms PDEN, LDSDG, and other DG algorithms by 2% to 6% in overall accuracy.Its coarse- and fine-grained text representations support visual-linguistic alignment rather than the data-generation strategies used by PDEN and LDSDG.
  • Classification maps: LDGnet produces less noisy and more accurate classification maps in several areas, including Garden land and Lake in GID-wh.Brick and bare soil in Pavia Center are also reported as greatly improved compared with the comparison methods.
  • Computational cost: LDGnet has relatively high computational cost because its three-layer transformer text encoder contains 33.43M parameters, or 97.7% of the total.Its overall computational complexity is nevertheless reported as lower than LDSDG's.

V. CONCLUSIONS

The paper concludes that LDGnet combines visual and linguistic modalities to learn cross-domain invariant representations from prior linguistic knowledge. Coarse- and fine-grained text representations guide this process through a shared semantic space.

  • Conclusion: LDGnet uses a dual-stream architecture to extract visual and linguistic features concurrently.The linguistic features provide prior knowledge treated as cross-domain shared knowledge for guiding invariant representation learning.
  • Conclusion: Coarse-grained and fine-grained text representations guide cross-domain invariant representation learning through visual-linguistic alignment.The text representations are used as prior knowledge in the shared semantic space.
Loading 2209.02700v1…