Source-linked AI summary
Few-Shot Hyperspectral Image Classification With Unknown Classes Using Multitask Deep Learning
Shengjie Liu, Qian Shi, Liangpei Zhang
TL;DR
Closed-world hyperspectral classifiers must assign unrecognized samples to known classes, which can overestimate land-cover areas. MDL4OW combines classification and reconstruction, using reconstruction loss and extreme value theory to identify unknowns. Experiments reported improved performance, especially in few-shot settings, including a 4.94% overall-accuracy improvement on Salinas.
Problem
Closed-world hyperspectral classification assumes all unseen samples belong to predefined classes, although real scenes may contain overlooked novel classes that are consequently assigned known labels.
Method
MDL4OW jointly performs classification and reconstruction, comparing original and reconstructed data and applying extreme value theory to separate known from unknown samples.
Results
MDL4OW achieved the best University of Pavia few-shot classification across OA, F1, and mapping error, with values of 85.07%, 0.9172, and 14.03%.
Takeaways & Limitations
Considering unknown classes improves hyperspectral image classification accuracy, particularly when training samples are scarce.
Abstract
from arXiv · showhide
Current hyperspectral image classification assumes that a predefined classification system is closed and complete, and there are no unknown or novel classes in the unseen data. However, this assumption may be too strict for the real world. Often, novel classes are overlooked when the classification system is constructed. The closed nature forces a model to assign a label given a new sample and may lead to overestimation of known land covers (e.g., crop area). To tackle this issue, we propose a multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (named MDL4OW) where unknown classes may exist. The reconstructed data are compared with the original data; those failing to be reconstructed are considered unknown, based on the assumption that they are not well represented in the latent features due to the lack of labels. A threshold needs to be defined to separate the unknown and known classes; we propose two strategies based on the extreme value theory for few-shot and many-shot scenarios. The proposed method was tested on real-world hyperspectral images; state-of-the-art results were achieved, e.g., improving the overall accuracy by 4.94% for the Salinas data. By considering the existence of unknown classes in the open world, our method achieved more accurate hyperspectral image classification, especially under the few-shot context.
I. INTRODUCTION
Hyperspectral classification typically assumes a closed, complete label system, but real scenes can contain unannotated classes that are forced into known categories. MDL4OW addresses this open-world, few-shot setting by combining classification, reconstruction, and extreme-value-based unknown detection.
- Unannotated buildings, swimming pools, vehicles, and parking sheds in University of Pavia imagery were misclassified as natural land covers or unrelated materials.
- Closed-world classifiers assign unknown instances to known labels, potentially overestimating mapped crop areas and food supplies.
- Few-shot hyperspectral datasets provide far fewer labeled samples than common image-recognition benchmarks, limiting centroid-based open-set methods.
- MDL4OW jointly performs classification and reconstruction, using reconstruction discrepancies to estimate whether samples are unknown without class-wise centroid estimation.
- The method uses extreme value theory with all data to estimate unknown scores and proposes mapping error for imbalanced open-world evaluation.
II. METHODOLOGY
The proposed network extracts spectral-spatial features, classifies them with SoftMax, and incorporates an encoder-decoder reconstruction branch. Its classification component is based on convolutional feature extraction and cross-entropy optimization.
- MDL4OW uses an encoder with residual learning blocks and global average pooling, followed by a fully connected SoftMax classifier.
- Convolutional layers extract spectral-spatial representations from hyperspectral sample instances for subsequent classification.
- The classifier converts latent features into probabilities for the predefined categories and is trained by minimizing cross-entropy loss.
- The flowchart specifies a 9 × 9 × Channel input and a reconstruction output with the same dimensions.
- A fixed SoftMax threshold of 0.5 is used as a baseline for rejecting low-confidence instances as unknown.
B. Reconstruction via multitask learning
The multitask network adds reconstruction to classification so that samples poorly represented by labeled latent features can be rejected as unknown.
- A reconstruction task is added because the classification network alone cannot identify unknown instances.
- The decoder reconstructs each input from encoder latent features, and reconstruction quality is measured with ℓ1 distance.
- Training minimizes a combined classification and reconstruction objective through backpropagation.
- Transposed convolution layers progressively expand 1×1×channel latent features into patch-based hyperspectral instances.
C. Threshold setting with extreme value theory
MDL4OW identifies unknown samples from the upper tail of reconstruction losses and uses extreme value theory to set a separation threshold. The method supports both global and class-wise variants, with tail size controlling the fitted distribution.
- Larger reconstruction losses indicate instances insufficiently optimized by the network, so the loss tail is treated as unknown.
- Extreme value theory models the reconstruction-loss tail to determine a boundary between known and novel samples.
- The tail is modeled with a Weibull-related extreme-value formulation and generalized Pareto distribution.
- Tail size τ and threshold z determine the known-versus-novel decision, with τ required to be at least two for distribution modeling.
- MDL4OW estimates the unknown score without class centroids, while MDL4OW/C provides a class-wise comparison variant.
III. DATASETS AND EXPERIMENTAL SETUP
The experiments use real hyperspectral datasets containing annotated known classes and manually identified unknown materials, while MDL4OW combines classification, reconstruction, and EVT-based unknown detection.
- Datasets: The University of Pavia image contains 103 bands, nine standard classes, and six manually annotated additional classes.The scene is 610×340 pixels at 1.3 m GSD after removing 12 noisy or water-absorption bands.
- Method: MDL4OW extracts latent features, predicts known classes, computes reconstruction error, and applies an EVT model to identify unknowns.The algorithm compares reconstruction error with an EVT model and threshold before producing the predicted label.
- Datasets: The Salinas Valley dataset contains 16 agricultural land covers, with man-made materials and a water pool manually annotated as unknown.It covers 512×217 pixels and retains 204 bands after discarding 20 water-absorption bands.
- Datasets: The Indian Pines image contains 16 annotated classes, while a road outside the known classes was manually annotated as unknown.Some classes have fewer than ten instances, illustrating the few-shot setting.
B. Evaluation metrics
Open-world evaluation extends classification metrics to include unknown classes, but class imbalance can make open overall accuracy reflect unknown-class performance more than known land-cover accuracy.
- Openness: Openness measures the degree to which a dataset includes unknown or open-world content.The reported openness values for three hyperspectral datasets are 13%, 3%, and 20%.
- Open overall accuracy: Open-world evaluation must account for unknown classes rather than treating the classification system as closed.A naive approach extends overall accuracy from C classes to C + 1 classes by adding the unknown class.
- Open overall accuracy: When known classes occupy a small portion of the data, open overall accuracy can be dominated by unknown-class performance.The paper therefore distinguishes interest in known land-cover accuracy from the open metric’s inclusion of unknown accuracy.
- Open overall accuracy: Open overall accuracy resembles closed overall accuracy but includes the unknown class.The metric therefore evaluates both known and unknown classes together.
3) Micro F1 score:
The paper uses micro F1 to emphasize known-class precision and recall, and proposes mapping error to assess area-estimation accuracy under unknown classes and imbalance.
- Micro F1 score: Micro F1 considers precision and recall for known classes while excluding the unknown class’s F1 score.Unknown instances assigned to known classes reduce precision, making unknown-class handling relevant to the metric.
- Mapping error: Mapping error compares predicted and ground-truth class areas across known classes and the unknown class.Ap,i denotes predicted area, Agt,i ground-truth area, C the number of known classes, and C+1 the unknown class.
- Mapping error: In the closed setting, mapping error ranges from 0–2, while the open-world maximum depends on the evaluated unknown area.The paper gives Errormax = 2 × (1 + Agt,C+1/ PC i=1 Agt,i).
- Mapping error: Mapping error is sensitive to imbalanced mapping and long-tail classes, which can produce land-cover area overestimation and underestimation.A model can achieve the same overall accuracy as another model while producing different mapping errors.
- Mapping error: Mapping error is not a point-by-point metric, so zero error does not necessarily imply zero classification confusion.It is presented as a simple, direct measure of land-cover area estimation, especially when unknown classes exist.
C. Experimental setup
Experiments evaluated MDL4OW under few-shot and many-shot settings across Pavia, Salinas, and Indian Pines, including known and novel classes. Results compared classification accuracy, F1, mapping error, and unknown-class identification.
- Experiments used 20 and 200 training samples per class to represent few-shot and many-shot settings.
- The evaluation included both almost-complete systems with few unknown instances and protocols treating unselected known classes as unknown.
- University of Pavia: 85.07% OA, 0.9172 F1, and 14.03% mapping error made MDL4OW best on three few-shot Pavia metrics.
- University of Pavia: 4.9% mapping-error reduction, from 12.19% to 7.29%, was achieved on Pavia.
- Unknown-class maps: The proposed methods identified non-annotated or non-agricultural materials as unknown in the Pavia, Salinas, and Indian Pines maps.
- Salinas Valley: 4.94% OA improvement, from 82.46% to 87.40%, was obtained on few-shot Salinas, alongside an F1 increase from 0.9038 to 0.9273.
- Indian Pines: 2.47% higher OA and 0.0104 higher F1 than closed classification were obtained by MDL4OW/C on many-shot Indian Pines.
4) Threshold analysis:
Threshold analyses examined unknown detection across datasets, classification-system completeness, and training-set size. Results show that confidence thresholding is insufficient, while performance depends on the openness and completeness of the class system.
- Threshold analysis: SoftMax confidence thresholding is insufficient because classifier uncertainty does not necessarily indicate an unknown class.
- Threshold analysis: MDL4OW is more robust to threshold selection than MDL4OW/C under the few-shot setting.
- Open-OA comparison: MDL4OW/C achieved 94.34% open OA on Salinas, while MDL4OW achieved 93.73% open OA and 96.37% closed OA.
- Imperfect systems: For incomplete classification systems, MDL4OW significantly improved F1, whereas improvements were less significant for other systems.
- Comparison: In most cases, the proposed method outperformed competing methods across the reported evaluation tables.
- Imperfect systems: MDL4OW/C can perform well in few-shot settings when openness is large enough to provide a sufficiently large tail for distribution estimation.
D. Reconstruction analysis
Reconstruction analysis examined whether known and unknown spectral profiles are represented differently by the learned model. Known profiles remained structured after reconstruction, whereas unknown profiles were poorly reconstructed.
- Known Pavia trees retained a classic vegetation curve after reconstruction, with tighter profiles indicating reduced intraclass variation.
- Unknown building-1 profiles were poorly reconstructed, supporting reconstruction quality as an unknown-class signal.
E. Sensitivity of the tail number
The tail number τ affects performance and must be selected carefully, especially for class-wise extreme-value modeling. MDL4OW was less sensitive to τ than MDL4OW/C in few-shot experiments.
- MDL4OW was less sensitive to tail number τ than MDL4OW/C under the few-shot setting.
- Training augmentation multiplied the samples per class by 4, easing extreme-value estimation under few-shot conditions.
- For Salinas, the F1-safe tail-number range was 2–12, with τ = 2 optimal.
- EVT does not work with τ = 1 because one instance cannot estimate a distribution.
- F1 can fall below baseline before OA because F1 evaluates known classes while OA includes the unknown class.
- MDL4OW achieved state-of-the-art results across OA, F1, and mapping error under the few-shot context.
F. Sensitivity of loss weights
Loss-weight sensitivity is evaluated with F1 score and mapping error under few-shot and many-shot settings across three datasets. Performance is generally stable, but few-shot results favor a classification weight of 0.7–0.8, while the Indian many-shot data respond differently.
- Evaluation: Figure 11 measures sensitivity using F1 score and mapping error for Pavia, Salinas, and Indian datasets under both sampling regimes.F1 score is better when larger, while mapping error is better when smaller.
- Few-shot setting: Few-shot performance degrades when the classification weight is below 0.2–0.3, with best results at 0.7–0.8.The analysis reports these trends across the examined settings and datasets.
- Many-shot setting: Many-shot performance is similar for Pavia and Salinas, whereas increasing the classification weight decreases F1 score for Indian data.
- Computation: MDL4OW and MDL4OW/C require more training and prediction time than HResNet because the reconstruction task adds computation.The two multitask variants share the same training procedure and training time.