Source-linked AI summary
Convolution-Free Holistic Multivariance Decomposition Layer for Efficient Hyperspectral Image Classification Tensor Networks
Süha Tuna, Ülker Başar
TL;DR
Hyperspectral image feature extractors struggle to capture complex interactions efficiently. This paper introduces convolution-free, end-to-end trainable HMD layers, with HMD-2 and HMD-1 achieving state-of-the-art performance at exceptional parameter efficiency.
Problem
Existing hyperspectral feature extractors struggle to capture complex interactions while maintaining low parameter cost.
Method
The HMD framework uses end-to-end trainable, convolution-free layers that isolate directional components and pairwise cross-domain interactions.
Results
97.85% and 96.40% mean Overall Accuracy were achieved by HMD-2 and HMD-1, respectively, with state-of-the-art classification performance.
Takeaways & Limitations
The reported discriminative advantage is achieved with exceptional parameter efficiency.
Takeaways & Limitations
The paper identifies unresolved structural bottlenecks in existing feature-extractor and EMPR implementations.
Abstract
from arXiv · showhide
Feature extraction for hyperspectral image classification is conventionally addressed using rigid tensor decompositions that fail to capture complex spatio-spectral interdependencies, or heavily parameterized convolutional neural networks that are computationally expensive. To overcome these limitations, this work introduces the Holistic Multivariance Decomposition (HMD) framework as a novel, end-to-end differentiable neural network layer. By explicitly separating independent single mode variations from cooperative higher dimensional interactions via learnable, matrix valued supports, the proposed HMD-0, HMD-1 and HMD-2 approximants are optimized jointly with a downstream classifier via backpropagation. Comprehensive evaluations across three benchmark HS datasets demonstrate that the higher level HMD layers achieve superior classification accuracy compared to classical learnable tensor baselines, including Tucker, Canonical Polyadic, and Tensor Train decompositions. Furthermore, HMD-1 and HMD-2 achieve a generalization capacity and training stability comparable to standard 2D and 3D-CNNs while requiring significantly fewer feature extractor parameters. These results demonstrate that the HMD framework provides a structurally robust substitute for traditional convolution in multidimensional HS image classification, offering high parameter efficiency and stability throughout the optimization process.
1 Introduction
HSI classification requires feature extractors that capture coupled spatial–spectral structure beyond the limits of conventional spectral, two-dimensional, and rigid tensor methods. This work makes HMD-0, HMD-1, and HMD-2 learnable layers and evaluates them against tensor and CNN baselines with emphasis on accuracy and parameter efficiency.
- Conventional spectral methods extract localized features effectively but struggle to capture HSI’s complex spatial correlations.These methods are applied along spectral signals, while their spatial modeling remains limited.
- Two-dimensional transform-based methods fail to simultaneously capture critical interband correlations across the spectral domain.Their inherent two-dimensional nature limits joint spatio-spectral representation.
- Rigid low-rank Tucker, CP, and TT decompositions often fail to capture complex mode interrelations, particularly joint spatio–spectral features.HDMR and EMPR isolate spatio-spectral correlations but lack flexible rank adjustment mechanisms.
- HMD extracts efficient spatio–spectral features while introducing a manageable dimensionality-reduction parameter for dynamically adjusting retained information.This addresses the limited flexibility of earlier multivariance frameworks.
- HMD-0, HMD-1, and HMD-2 are recast as fully differentiable feature-extraction layers whose support matrices are optimized jointly with a downstream classifier via backpropagation.The proposed layers are benchmarked against learnable Tucker, CP, and TT layers and 2D- and 3D-CNN baselines under a common protocol.
- The study evaluates classification performance and feature-extraction parameter cost across three benchmark HS datasets using shared classifiers and standardized metrics.Metrics include Overall Accuracy, Average Accuracy, macro-averaged F1-score, and Cohen’s kappa coefficient.
2 Background
Hyperspectral images contain strongly coupled spatial and spectral modes that conventional low-rank tensor decompositions can fail to preserve because their rank-one components model independent modal variations. HMD addresses this limitation with matrix-formed supports and full three-dimensional hierarchical components, while the broader comparison asks whether structured extractors can match CNN accuracy at lower parameter cost.
- Tensor decomposition background: HSI data cubes exhibit highly coupled spatial and spectral modes, requiring feature extraction methods that preserve spatio-spectral correlations.These structures motivate advanced multilinear algebra frameworks and computationally efficient tensor decompositions.
- Tensor decomposition limitations: Traditional low-rank approximations overlook coupled interdependencies because rank-one outer products capture only isolated variations of individual modes.This independence assumption sacrifices structural fidelity and fails to capture shared cross-mode dynamics important for robust feature extraction.
- Motivation for HMD: EMPR remains constrained by one-dimensional support structures, motivating a transition to matrix-formed supports that jointly preserve spatial–spatial and spatio–spectral interactions.The HMD framework is introduced to overcome this structural bottleneck.
- HMD framework: HMD distinguishes itself from HDMR and EMPR by retaining full three-dimensional hierarchical components and exposing a single dimensionality-reduction parameter r.The parameter r controls the trade-off between subspace compression and retained discriminative information.
3 Methods
The methods represent hyperspectral patches as third-order tensors and compare learnable, differentiable tensor feature extractors under a unified end-to-end classifier formulation. HMD extends conventional decompositions by separating interaction degrees while preserving full tensor components and increasing truncation levels without adding feature-extractor parameters.
- Tensor representation: Hyperspectral patches are modeled as third-order tensors whose first two modes represent spatial extent and whose third mode represents spectral bands.Mode products contract a tensor mode against a factor matrix, and the compact Tucker operator applies three compatible mode matrices.
- Unified feature-extraction framework: All eight feature extraction methods are learnable, differentiable mappings trained end-to-end with an identical downstream classifier, isolating extractor effects in accuracy and parameter comparisons.The unified formulation prevents confounding from incidental architectural differences.
- Holistic Multivariance Decomposition: HMD separates baseline, independent single-mode, and pairwise cross-mode variations using matrix-valued supports that capture spatial-spatial and spatio-spectral interactions.Unlike vector-valued supports, the framework preserves spatial and spectral structures within each isolated component as full three-dimensional tensors.
- Parameter efficiency: Higher HMD truncation levels reuse identical trainable support matrices, changing only the pooled hierarchical components and therefore not increasing feature-extractor parameters.This shared-support structure enables higher-order approximants to achieve performance gains at an identical parameter count.
4 Experimental Setup
The experiments compare HMD, tensor decomposition, and convolutional feature extractors across Indian Pines, Pavia University, and Loukia using standardized spatio-spectral patches and a shared classifier. Evaluation combines classification metrics with trainable-parameter accounting to assess accuracy relative to feature-extraction cost.
- Datasets: Three benchmark HS datasets cover agricultural, urban, and mixed land-cover settings: Indian Pines, Pavia University, and Loukia.Loukia extends evaluation beyond the conventional Indian Pines–Pavia University pairing to assess generality and robustness.
- Compared methods: The 2D-CNN used 8 and 16 channels with 3 × 3 kernels, whereas the 3D-CNN used 4 and 8 channels with 3 × 3 × 3 kernels.Filter counts and decomposition ranks remained fixed across datasets to isolate feature-extractor structure.
- Evaluation: Held-out performance was assessed using OA, AA, F1, and Cohen’s κ, alongside total and feature-extractor-specific trainable-parameter counts.This enables direct comparison of classification accuracy relative to parametric cost.
5 Results
Across three hyperspectral datasets, HMD-1 and HMD-2 achieved the strongest classification results, with HMD-2 attaining the highest mean metrics and HMD-1 delivering the best Indian Pines performance. Higher-order HMD layers also showed greater rank robustness and training stability than classical tensor decompositions while using fewer parameters than CNN baselines.
- Classification performance: HMD-2 achieved the highest mean OA of 97.85%, AA of 96.51%, F1 of 0.9703, and κ of 0.9742 across the three datasets.HMD-1 followed, while 2D-CNN ranked third and 3D-CNN fourth on all four metrics.
- Baseline comparisons: HMD-2 exceeded HMD-0 by up to 20.4 and 21.6 OA percentage points on Pavia University and Loukia, respectively.The gap narrowed on Indian Pines, where CP remained competitive with OA = 74.93%.
6 Conclusion
The HMD framework provides end-to-end trainable, convolution-free feature extraction for hyperspectral image classification, isolating directional and cross-domain interactions. HMD-1 and HMD-2 deliver high accuracy with parameter efficiency, structural robustness, and stability across challenging training conditions.
- Framework: HMD implements end-to-end trainable, convolution-free hyperspectral feature-extraction layers that isolate linear directional components and pairwise cross-domain interactions.HMD-1 and HMD-2 are presented as addressing structural bottlenecks in rigid tensor decompositions.
- Performance: 97.85% and 96.40% mean Overall Accuracies are achieved by HMD-2 and HMD-1, respectively, across agricultural, urban, and mixed land-cover scenes.The conclusion characterizes these results as consistently state-of-the-art classification performance.
- Efficiency: HMD methods match baseline core tensor models’ minimal footprint while requiring substantially fewer trainable weights than conventional CNN architectures.The reported discriminative advantage is therefore paired with exceptional parameter efficiency.
- Robustness: HMD’s discriminative advantage extends beyond a specific sensor modality or class distribution and is strongest with limited labeled samples and high class counts.The framework also insulates latent features from optimization noise under constrained ranks or limited training regimes.
- Conclusion: HMD is characterized as a computationally lightweight, stable, and interpretable deep-learning layer offering an efficient alternative to standard convolutional networks.The conclusion positions the framework for advanced hyperspectral classification.