PaperlayerView exact revision on arXiv

Source-linked AI summary

Turning spectra into images improves plant trait retrieval with 2D-CNNs

Javier Lopatin, Teja Kattenborn, Eya Cherif, Sebastián Moreno

arXiv:2608.16661v1cs.CVcs.LG

TL;DR

One-dimensional processing may miss long-range relationships among spectral bands needed for plant-trait prediction. This study compares 2D spectral encodings and self-supervised pretraining, finding that a direct reshape yields the strongest multi-trait performance.

  • Problem

    1D-CNNs process spectra as flat sequences, limiting their ability to capture long-range inter-band dependencies relevant to plant-trait signatures.

  • Method

    The study compares nine 1D-to-2D spectral transformations and 2D masked-autoencoder pretraining against established 1D approaches for multi-trait retrieval.

  • Results

    R2 = 0.684 for direct reshaping improved all eight traits over the 1D baseline by +0.097, while 2D self-supervision outperformed 1D counterparts in-distribution.

  • Takeaways & Limitations

    The gain is attributed to the 2D representation, which lets standard convolutional kernels integrate spectrally distant bands without relying on elaborate encodings or ImageNet weights.

  • Takeaways & Limitations

    Under cross-dataset shift, 2D masked-autoencoder fine-tuning transferred worse than 1D self-supervision, and all models lost substantial accuracy.

Abstract

from arXiv · show

Hyperspectral reflectance spectroscopy enables non-destructive estimation of plant functional traits, yet current deep learning approaches process spectra as one-dimensional sequences, which limits how they capture long-range inter-band dependencies. We asked whether transforming 1D spectra into 2D image representations improves multi-trait prediction with convolutional neural networks (CNN). We compared nine transformations using EfficientNet-B0 on the GreenHyperSpectra dataset (7,897 labeled spectra, eight traits, 400-2450 nm), benchmarked against published 1D CNN results on the same split. Trained from scratch, the simplest transformation, a direct Reshape of the spectrum into a 2D grid, performed best ($R^2 = 0.684 \pm 0.001$) and improved on the state-of-the-art 1D baseline ($R^2 = 0.587$, $+0.097$). We then pretrained a 2D masked autoencoder (MAE-2D) on 139,000 unlabeled spectral images. Linear probing, which freezes the encoder and trains only a multilayer perceptron head, reached $R^2 = 0.646$ and exceeded every 1D self-supervised counterpart, including the fine-tuned MAE-1D ($R^2 = 0.641$). Under cross-dataset evaluation all models lost most of their accuracy and none beat the 1D baseline significantly. To identify which wavelengths drive each prediction, we applied Integrated Gradients and Grad-CAM and unfolded band importance back to the spectral axis. Protein ($r = 0.45$) and leaf water ($r = 0.33$) agreed with sensitivities simulated by the PROSAIL radiative-transfer model, while carotenoids ($r = 0.06$) and leaf area index ($r = -0.11$) did not, showing that the model reads established leaf chemistry for traits with sharp absorption features. The representational advantage of 2D spectral images, rather than architectural complexity or ImageNet pretraining, drives the gain over 1D approaches.

1. Introduction

Hyperspectral spectroscopy offers scalable, non-destructive plant-trait estimation, but 1D-CNNs may miss dependencies among absorption features distributed across the spectrum. This study systematically evaluates 1D-to-2D spectral encodings, 2D self-supervised pretraining, transferability, and wavelength-importance visualization for trait retrieval.

  • Motivation: Hyperspectral reflectance spectroscopy estimates plant functional traits non-destructively because foliar biochemistry and canopy structure create characteristic absorption features across 400–2500 nm.These traits support understanding ecosystem functioning, biodiversity, carbon cycling, photosynthesis, light interception, and nutrient cycling.
  • Motivation: 1D-CNNs process spectra as flat sequences, limiting receptive fields to local bands despite trait signatures arising from interactions across the full wavelength range.Examples include combined pigment, water, and structural-scattering effects across visible, shortwave-infrared, and red-edge regions.
  • Related approaches: Direct reshaping folds a 1D spectrum into a 2D matrix while preserving wavelength adjacency as spatial texture, whereas alternative layouts and pairwise matrices encode spectral relationships differently.Alternating raster and Hilbert-curve layouts preserve adjacency more faithfully, while normalized-difference matrices encode every band pair.
  • Study aims: Nine 1D-to-2D transformation methods are systematically compared as spectral encodings for 2D-CNN-based plant-trait retrieval.Prior work has mainly examined classification rather than continuous regression, and lacked controlled comparisons of multiple transformations.
  • Study aims: 2D representations are evaluated for masked-autoencoder pretraining, transferability to unseen sites, and visualization of variable importance.The study also examines whether 2D pretraining improves predictions beyond 1D pretraining.

2. Methods · 2.1. Study Data · 2.2. 1D-to-2D Spectral Transformations

The study used GreenHyperSpectra’s labeled and unlabeled hyperspectral data to compare nine ways of converting 1D spectra into 224×224 image inputs. These transformations ranged from spatial rearrangements and time-frequency representations to correlation, transition, interaction, and composite encodings.

  • 2.1. Study Data: GreenHyperSpectra contains 7,897 labeled canopy spectra and 139,295 unlabeled spectra collected across platforms, continents, ecosystems, and sensor contexts.The labeled data include co-located functional-trait measurements, while the unlabeled data support representation learning.
  • 2.1. Study Data: 400–2450 nm spectra provide one reflectance value per wavelength as the input signal.The wavelength range defines the spectral axis subsequently encoded by the image transformations.
  • 2.1. Study Data: The labeled subset spans 50 field campaigns and seven PROSAIL-PRO-related traits, with missing measurements handled through masked loss functions.Trait values were harmonized to area-based units, but not every sample contains every trait measurement.
  • 2.2. 1D-to-2D Spectral Transformations: Nine transformation methods encoded different spectral information into spatial image structure, producing 224×224 images with varying channel counts.The outputs were designed for standard convolutional neural network architectures.
  • 2.2.1. Direct Reshape: Direct Reshape folds 1,721 bands into a zero-padded 42×42 matrix, then resizes and independently min-max normalizes it to 224×224.The transformation appends 43 zeros and preserves each image’s own spectral scaling.
  • 2.2.2. Serpentine and Hilbert-Curve Reshape: Serpentine and Hilbert-curve variants address Reshape’s row-boundary discontinuities by reorganizing consecutive bands into spatially adjacent pixels.Direct Reshape preserves global wavelength order but breaks adjacency at each row boundary.
  • 2.2.3. Continuous Wavelet Transform: The Continuous Wavelet Transform uses a Morlet wavelet across 128 logarithmic scales to form a 128×1,721 scalogram capturing multiscale spectral structure.The representation is computed with PyWavelets from the absolute wavelet coefficients.

2.3. Model Architectures

The study framed plant-trait prediction as a single eight-output regression task using 2D spectral images. It compared supervised EfficientNet-B0 training from scratch with self-supervised 2D masked-autoencoder pretraining followed by fine-tuning.

  • Model Architectures: One model jointly predicted all eight traits rather than training eight separate models.The task was treated as single multi-output regression.
  • Model Architectures: Supervised regression used EfficientNet-B0 as a roughly 4 M-parameter backbone trained from scratch.The architecture was selected for its balance of model capacity and computational efficiency with n = 4,508 labeled spectra.
  • Model Architectures: The self-supervised model was a 2D masked autoencoder pretrained on 139,295 unlabeled spectra represented as Reshape images.This enabled standard vision-transformer components for representation learning.
  • Model Architectures: The MAE used a ViT encoder with 16 × 16 patches, 192-dimensional embeddings, six blocks, three heads, and 75% random masking.Only visible patches were encoded, while a lightweight decoder reconstructed masked patch pixels; the full model comprised approximately 3.3 M parameters.

2.4. Training Protocol

Models used a standardized, stratified GreenHyperSpectra split and were trained with masked losses to accommodate sparse trait labels. Supervised EfficientNet-B0 and MAE-2D training used distinct optimization schedules, validation procedures, and early stopping protocols.

  • Data split: The standardized split assigned 4,508 spectra to training and 1,127 to testing, stratified by source dataset.The split preserved proportional representation of vegetation types, sensors, and acquisition conditions.
  • Supervised training: Supervised EfficientNet-B0 models used AdamW with β1 = 0.9, β2 = 0.999, weight decay = 10^-4, and cosine annealing from 10^-3 to 10^-6.Training allowed up to 100 epochs with early stopping at patience = 15 epochs.
  • Loss and augmentation: Masked MSE computed gradients only for observed trait values, while augmentation applied additive baseline shifts uniformly from ±2% of the spectral range.The label mask was distinct from random patch masking of the input image.
  • MAE pretraining: MAE-2D pretraining used 138,920 valid Reshape images from 139,295 unlabeled spectra, with AdamW at lr = 10^-4 for 300 epochs.Training used batch size = 32, gradient clipping max norm = 1.0, a 95/5 train/validation split, and early stopping at patience = 30 epochs.
  • MAE fine-tuning: After pretraining, the decoder was discarded and the encoder was fine-tuned jointly with a 192 →192 →8 MLP regression head using lr = 10^-4.Fine-tuning followed supervised training with AdamW, early stopping, masked MSE loss, and 100 maximum epochs.

2.5. Experimental Design

The experimental design comprises three complementary case studies evaluating transformation choice, self-supervised pretraining, and generalization to unseen field campaigns. Together, these studies compare supervised 2D models with 1D baselines and assess performance across in-distribution and cross-dataset settings.

  • Overall design: Three complementary case studies evaluate 2D spectral representations through transformation comparison, self-supervised pretraining, and out-of-distribution generalization.The studies progressively examine representation choice, pretraining strategy, and performance on unseen field campaigns.
  • Case Study 1: Transform comparison: Nine 1D-to-2D transformations were evaluated with EfficientNet-B0 under identical supervised training conditions against Cherif et al.’s supervised EfficientNet-1D baseline (mean R2 = 0.587 across eight traits).This was the transformation-comparison case study.
  • Case Study 2: Self-supervised pretraining: MAE-2D pretraining on 139K unlabeled images was compared with supervised training without pretraining and Cherif et al.’s MAE-1D (R2 = 0.641).The pretrained models were assessed using the best transformation methods and fine-tuned on real labeled data.
  • Case Study 3: Out-of-distribution generalization: Cross-dataset evaluation tested generalization to unseen field campaigns by holding out five labeled sub-datasets from 50 and repeating the protocol over 12 folds.Each fold trained on the remaining datasets with an internal 80/20 training-validation split, and every dataset was held out at least once.

2.6. Evaluation Metrics

Model performance was evaluated per trait on a held-out test set using R2 and nRMSE, with results summarized across three random seeds and averaged across traits for method-level comparison. Transform significance was tested against the published 1D baseline using a paired Wilcoxon signed-rank test across eight traits.

  • Performance metrics: R2 and nRMSE were computed per trait on the held-out test set.nRMSE was expressed as a percentage of the observed values’ 1st–99th percentile range.
  • Performance metrics: Mean and standard deviation were reported across three random seeds.
  • Performance metrics: Per-trait R2 values were averaged into one summary statistic per method for comparison with Cherif et al. (2025) baselines.
  • Significance testing: Significance was assessed against Cherif et al. (2025)’s reported 1D baseline with a paired Wilcoxon signed-rank test on per-trait R2 differences across eight traits.The non-parametric paired test was chosen because traits are jointly predicted and not independent.

2.7. Spectral Importance

The study quantified spectral importance for the best Reshape + EfficientNet-B0 model using Integrated Gradients and Grad-CAM. Integrated Gradients was unfolded into a 1D wavelength profile and compared with PROSAIL-derived sensitivities, while Grad-CAM remained a 2D region-level attribution.

  • Attribution methods: Integrated Gradients and Grad-CAM independently attributed predictions from the best supervised Reshape + EfficientNet-B0 model.Integrated Gradients provides signed input-level attributions, whereas Grad-CAM provides region-level relevance over the final convolutional embedding.
  • Attribution methods: The Reshape mapping preserved band order, allowing Integrated Gradients to be unfolded into a 1721-band wavelength importance profile.The mapping was pixel (i, j) to band k = 42i+j, making the resulting profile directly comparable to classical band selection.
  • Attribution methods: Grad-CAM remained two-dimensional because its 7 × 7 convolutional embedding localizes regions and aliases into a comb pattern on the fine band axis.The two attribution methods were applied independently, so neither method served as input to the other.
  • Physical reference: Empirical band importance was compared with a physically grounded PROSAIL radiative-transfer reference by sweeping each corresponding leaf or canopy parameter across 25 steps.Remaining leaf parameters were held at their GreenHyperSpectra mean, while structure, soil, and geometry parameters were fixed to the stated simulation values.

3. Results

On GreenHyperSpectra, 2D spectral representations improved multi-trait retrieval over 1D baselines in supervised and self-supervised settings, led by direct Reshape. However, this advantage largely disappeared under cross-dataset evaluation, while wavelength importance aligned with PROSAIL for some traits but only modestly overall.

  • Supervised 2D transformations: +0.036 to +0.174 were the per-trait gains from Reshape over the 1D baseline, with the largest improvements for anthocyanins, carotenoids, and carbon-based constituents.Anthocyanins reached R^2 = 0.628 with ΔR^2 = +0.174, while carotenoids reached R^2 = 0.691 with ΔR^2 = +0.147.
  • Supervised 2D transformations: R^2 = 0.684 ± 0.001, with Reshape, Serpentine, Hilbert, the multi-window spectrogram, and CWT significantly outperforming the 1D baseline at R^2 = 0.587.Each of the five transformations won seven to eight of the eight traits.
  • Self-supervised representations: R^2 = 0.646 ± 0.020 was achieved by linear probing of MAE-2D, exceeding the MAE-1D fine-tuning result at R^2 = 0.641.The frozen encoder and regression head captured 97% of fine-tuned MAE-2D performance, which was R^2 = 0.667.
  • Cross-dataset evaluation: R^2 = 0.333 was the best cross-dataset result, achieved by the five-channel Reshape+CWT+NDI composite, while plain Reshape fell to R^2 = 0.305.Every transformation lost substantial in-distribution accuracy and the ranking changed; MAE-2D fine-tuning reached only R^2 = 0.153 versus MAE-1D at R^2 = 0.311.
  • Spectral interpretation: r = 0.16 was the mean agreement between Integrated Gradients importance and PROSAIL sensitivity, with strongest correlations for protein at r = 0.45 and leaf water at r = 0.33.Carotenoids showed r = 0.06 and leaf area index showed r = -0.11.

4. Discussion

Discussion attributes the performance gain primarily to the 2D representation, whose multi-scale receptive fields integrate distant spectral bands and improve all eight traits. Wavelength attribution supports chemically meaningful predictions for several traits, while cross-dataset transfer remains limited and self-supervised pretraining contributes less than reshaping.

  • Representation advantage: 0.684 mean R2 versus 0.587 for the supervised 1D-CNN shows that direct Reshape improved multi-trait retrieval across all eight traits and outperformed the nine tested transformations.Reshape also achieved the lowest variance among layouts.
  • Representation advantage: A 3×3 kernel in the 42×42 image spans about 84 nm, versus about 3 nm for an equivalent 1D kernel, enabling later layers to combine features across hundreds of nanometers.This receptive field can jointly capture pigment absorption near 680 nm, the red-edge near 750 nm, and shortwave-infrared structural features.
  • Trait-specific gains: +0.174 for anthocyanins and +0.147 for carotenoids were the largest trait gains, followed by +0.110 for carbon-based constituents and +0.106 for leaf mass per area.These traits rely on distributed signals, correlated bands, or multiple shortwave-infrared windows, whereas chlorophyll and leaf water gained +0.080 and +0.082.
  • Wavelength attribution and transfer: Protein (r = 0.45) and leaf water (r = 0.33) showed the strongest agreement with PROSAIL sensitivity, with importance near diagnostic absorption features.Seven of eight traits agreed with PROSAIL overall, while cross-dataset transfer was better for water and dry-matter traits than for carotenoids and leaf area index.
  • Pretraining and generalization: 0.097 was the supervised gain from moving from 1D to 2D, compared with 0.054 from pretraining a 1D masked autoencoder on 139,000 unlabeled spectra.MAE-2D fine-tuning reached R2 = 0.667 ± 0.006 and beat MAE-1D fine-tuning by 0.026, but dropped to 0.153 under cross-dataset shift.
  • Ecological implications: +0.110 and +0.106 for carbon-based constituents and leaf mass per area improve retrieval of dry-matter traits relevant to carbon stocks, decomposition, nutrient cycling, and the leaf economics spectrum.A single model improving all eight traits also preserves covariance structure used by functional diversity metrics such as FDis, FRic, FEve, and Rao’s Q.

5. Conclusion

Turning one-dimensional hyperspectral reflectance into two-dimensional images improved multi-trait retrieval with CNNs, with the gain attributed to representation rather than architectural complexity or pretraining. The simple reshape also outperformed one-dimensional counterparts under self-supervised pretraining and offers a reproducible option for operational trait monitoring.

  • Conclusion: 0.684 R2 was achieved by directly reshaping spectra, the best of nine transformations, while improving all eight traits over the one-dimensional baseline by +0.097.The reshape also outperformed every more elaborate encoding.
  • Conclusion: The performance advantage came from the two-dimensional representation itself rather than architectural complexity or pretraining.
  • Conclusion: Under self-supervised pretraining, the two-dimensional representation outperformed its one-dimensional counterpart, and a frozen encoder with a lightweight linear head beat every one-dimensional baseline.This points to value where labeled data are scarce.
  • Conclusion: A reshape paired with a standard network is simple, stable, and reproducible for operational pipelines converting spectra into trait distributions for global monitoring.

CRediT authorship contribution statement

Javier Lopatin led the paper’s conception, execution, analysis, data curation, visualization, and drafting, while Teja Kattenborn, Eya Cherif, and Sebastián Moreno contributed to manuscript review and editing.

  • Javier Lopatin handled conceptualization, methodology, software, formal analysis, investigation, and data curation.
  • Javier Lopatin completed visualization, wrote the original draft, and conducted writing review and editing.
  • Teja Kattenborn, Eya Cherif, and Sebastián Moreno contributed to writing review and editing.

Funding

The work was supported by Chile’s Agencia Nacional de Investigación y Desarrollo (ANID) through grant 11241088, which had no role in the study or publication decision.

  • Funding: ANID supported the work through Chilean grant 11241088, without involvement in study design, data handling, reporting, or the decision to submit.The funder had no role in data collection, analysis, interpretation, or writing.

Supplementary Data

The supplementary material provides error-based comparisons across transformations, traits, training regimes, and cross-dataset evaluation. It also documents attribution analyses linking 2D relevance maps and unfolded spectral importance to PROSAIL sensitivity.

  • In-distribution error comparisons: Tables S1–S3 report in-distribution nRMSE (%) for transformations, per-trait comparisons, and training regimes against the supervised 1D baseline, with lower values indicating better performance.The tables use mean ± standard deviation over three seeds where specified and report differences from the 1D baseline.
  • Cross-dataset evaluation: Tables S4–S6 report cross-dataset nRMSE (%) and per-trait R2 under leave-datasets-out evaluation, including supervised and self-supervised 1D-to-2D comparisons.Cross-dataset 2D results use the Reshape transformation, with single-run values for the reported comparisons.
  • Attribution and sensitivity: Seven of eight traits show positive alignment between empirical band importance and PROSAIL theoretical spectral sensitivity.Figure S2 quantifies agreement using Pearson r and Spearman ρ across all 1721 bands.
  • Attribution and sensitivity: Figures S3–S10 visualize Grad-CAM relevance on Reshape images and link 2D attribution maps to unfolded 1D importance profiles for the eight traits.Figures S4–S10 cover the seven traits beyond the overview shown in Figure S3.
Loading 2608.16661v1…