PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Multimodal Models for Environmental Hyperspectral Unmixing

Michał Cholewa, Luca Ciampi, Nicola Messina, Przemysław Głomb, Giuseppe Amato

arXiv:2609.01289v1cs.CV

TL;DR

Hyperspectral unmixing must recover endmembers and abundances from mixed pixels despite unknown scene composition and errors in modular pipelines. This paper introduces an LVLM agent that iteratively refines those decompositions using spectral-library retrieval, abundance-map inspection, and merge/discard actions. Across multiple datasets and pipelines, refinement generally improves decomposition quality and remains competitive with integrated methods, subject to computational and action-space limitations.

  • Problem

    Unknown endmember cardinality, signatures, and abundances make modular hyperspectral unmixing difficult, while errors and redundancies in candidate components affect the recovered decomposition.

  • Method

    An algorithm-agnostic LVLM agent refines modular decompositions by combining spectral-library evidence and abundance-map visualization, then merging or discarding components and re-estimating abundances.

  • Results

    Cardinality reductions were accompanied by lower mSAD in 22 of 24 comparisons, while aRMSE improved in 16 of 24 comparisons across pipeline configurations.

  • Takeaways & Limitations

    LVLM-guided refinement can improve endmember cardinality and recovered spectral and abundance representations across initializations while remaining competitive with integrated end-to-end unmixing methods.

  • Takeaways & Limitations

    Refinement is restricted to merge and discard operations, so the agent cannot add an independent material absent from the initial decomposition.

Abstract

from arXiv · show

Hyperspectral unmixing is a key task in remote sensing that aims to decompose mixed pixels in hyperspectral images into their constituent material signatures, or endmembers, and their fractional abundances. Conventional modular approaches estimate the scene composition through successive model-order estimation, endmember extraction, and abundance estimation stages, whose errors can lead to redundant or ambiguous candidate components and ultimately affect the recovered decomposition. We introduce an algorithm-agnostic, large vision-language model (LVLM)-driven agentic framework that refines the outputs of such pipelines rather than replacing their underlying numerical algorithms. Starting from an initial decomposition, the agent iteratively gathers complementary spectral and spatial evidence through dedicated tools, including spectral-library retrieval and abundance-map visualization, and modifies the active endmember set through merge and discard operations followed by abundance re-estimation. We apply the same refinement procedure to several modular pipelines combining different model-order, extraction, and abundance-estimation methods, and evaluate it on HYDICE Urban, Jasper Ridge, and Stonewall Playa. Experiments show that the proposed agent consistently improves endmember cardinality and generally improves the recovered spectral signatures and abundance maps across heterogeneous modular pipelines, while remaining competitive with integrated end-to-end unmixing methods, including CNN-AE, uDAS, and R-CoNMF. These results highlight the potential of tool-using LVLM agents to combine spectral and spatial evidence for algorithm-agnostic refinement of physically grounded hyperspectral unmixing decompositions. Code is publicly available at https://anonymous.4open.science/r/agentic-hu.

1. Introduction

Hyperspectral unmixing decomposes mixed pixels into material signatures and abundances, but unknown endmember counts and errors in modular stages make the task challenging. The proposed LVLM agent refines modular decompositions using complementary spectral and spatial evidence across multiple pipelines and datasets.

  • Hyperspectral unmixing decomposes mixed pixels into endmembers and their fractional abundances.
  • Unknown endmember counts, signatures, and abundance maps complicate operational unmixing when scene-specific reference information is unavailable.
  • The framework refines initial modular-pipeline decompositions rather than replacing their underlying numerical algorithms.
  • A ReAct-style agent combines spectral-library retrieval with abundance-map visualization over an RGB composite to select refinement actions.
  • The agent applies merge and discard operations followed by abundance re-estimation across different combinations of model-order, extraction, and abundance-estimation methods.
  • Evaluation covers HYDICE Urban, Jasper Ridge, and Stonewall Playa using multiple conventional initial pipelines.

2. Related Work

Related work includes modular, integrated, library-based, and foundation-model approaches to hyperspectral unmixing and remote-sensing reasoning. This framework positions an LVLM as an algorithm-agnostic controller that refines modular decompositions using retrieval and image-processing tools.

  • Modular unmixing pipelines combine model-order estimation, endmember extraction, and abundance estimation, with methods such as HySime, NWHFC, VCA, FCLSU, and UnDIP.
  • Integrated methods estimate signatures and abundances while progressively deactivating redundant components, but require an upper bound Kmax in advance.
  • Sparse and library-based methods select spectra from external dictionaries, where library coherence and scene mismatch motivate pruning and mismatch-aware approaches.
  • Hyperspectral foundation models learn spatial–spectral representations, while related multimodal systems support annotations and spatial–spectral reasoning.
  • Remote-sensing agents use language-model controllers, retrieval, and specialized tools primarily for image understanding and geospatial analysis.
  • The proposed controller retrieves reference spectra, inspects abundance maps, updates candidate endmembers, and triggers abundance re-estimation without replacing numerical pipeline algorithms.

3. Methodology

The methodology combines conventional modular unmixing pipelines with an LVLM agent that iteratively refines candidate endmembers using complementary spectral and spatial evidence. Each accepted merge or discard updates the decomposition and triggers abundance re-estimation within the underlying pipeline.

  • Framework: The framework begins with an initial decomposition whose unknown endmember count, signatures, and abundance maps are refined rather than replacing the numerical unmixing algorithms.Initial decompositions are produced directly from the hyperspectral image by modular pipelines.
  • Initial Modular Unmixing Pipelines: The eight initial pipelines combine HySime or NWHFC, VCA or SISAL, and FCLSU or UnDIP for model-order, endmember, and abundance estimation.These configurations provide heterogeneous starting decompositions for evaluating algorithm-agnostic refinement.
  • Agentic Refinement: A ReAct-style agent autonomously selects candidates, tools, and refinement actions while maintaining active endmembers, material labels, and operation history.Available tools include library_search, compute_abundance, merge_endmembers, and discard_endmember.
  • Evidence Gathering: Spectral-library retrieval supplies ranked reference spectra and candidate material labels without inserting retrieved spectra into the endmember matrix or reconstructing the data.Matches are ranked using Spectral Angle Distance and serve as auxiliary semantic evidence.
  • Evidence Gathering: Abundance-map visualization reshapes a component’s abundance row into a heatmap overlaid on an RGB composite to expose its spatial support and distribution.This provides spatial evidence complementary to spectral-library retrieval.
  • State Updates: After merging redundant candidates or discarding unsupported ones, the framework updates the active set and re-estimates abundances with the estimator associated with pipeline π.Subsequent observations use the updated state, so accepted actions modify the numerical decomposition itself.

4. Experimental Setting

Experiments evaluate the LVLM-driven refinement framework across three hyperspectral benchmarks, repeated paired runs, modular initializations, and ablations. Results show broad improvements over initial decompositions and competitiveness with integrated methods, alongside substantial computational overhead.

  • Setup and Implementation Details: Experiments use Qwen-3.6 27B with temperature 0, and each experiment averages 10 paired runs from identical initial decompositions.The paired design compares refined and unrefined solutions under the same stochastic initialization.
  • Datasets: The evaluation covers HYDICE Urban, Jasper Ridge, and Stonewall Playa, representing urban, vegetated, and arid mineral-rich land-cover scenarios.An additional Samson evaluation is reported in supplementary material.
  • Results and Discussion: The agent reduces cardinality error across all pipeline configurations, with lower mSAD in 22 of 24 comparisons and lower rRMSE in 23 of 24.aRMSE improves in 16 of 24 comparisons, making abundance improvements less uniform than spectral and reconstruction improvements.
  • Results and Discussion: NWHFC-initialized refined pipelines achieve the best mSAD, aRMSE, and rRMSE on all three datasets and win 11 of 12 dataset–metric comparisons.They also achieve the best cardinality error on Jasper Ridge and Stonewall Playa.
  • Computational Demands and Practical Trade-offs: The framework requires around 25 minutes on one H100 for HYDICE Urban, compared with 15 minutes for uDAS.Its traceable evidence and actions may nevertheless suit critical, non-latency-sensitive analyses requiring human oversight.
  • Ablations: The combined spectral-library and abundance-map tools provide the most balanced performance, while Qwen-3.6 achieves the strongest overall controller-ablation profile.Images alone produce the best cardinality and spectral accuracy; the library alone produces the lowest abundance error.

5. Conclusion

The paper presents an algorithm-agnostic LVLM agent that iteratively refines modular hyperspectral unmixing decompositions using spectral and spatial evidence. Results indicate improved cardinality and representation quality across initializations, but refinement cannot introduce components absent from the initial decomposition.

  • The framework refines modular unmixing outputs rather than replacing the underlying numerical algorithms.
  • Iterative updates revise the active endmember set using complementary spectral and spatial evidence gathered through specialized tools.
  • Refinement is limited to merge and discard operations, so the agent cannot add an independent material component missing from the initial decomposition.

A. Evaluation Metrics

The evaluation uses cardinality, spectral, abundance, and reconstruction metrics to assess complete decompositions under possible cardinality mismatch. These metrics combine global compatibility with correspondence-based fidelity and penalize unmatched components.

  • Cardinality error measures the absolute difference between estimated and reference endmember counts, penalizing both missed and spurious components.Methods infer cardinality directly from the observed scene.
  • Mean spectral angle distance averages angular discrepancies over the complete rectangular reference–estimate cost matrix.The assignment-free score prevents poorly compatible estimates from being hidden by favorable subset matching.
  • Cardinality-aware abundance error uses a rectangular one-to-one Hungarian correspondence and explicitly represents unmatched reference and estimated components.This construction handles cardinality mismatch and penalizes missing or additional components through unmatched abundance mass.
  • Correspondence-conditioned reconstruction RMSE retains selected abundance rows and renormalizes them at each pixel before reconstruction.This prevents unmatched extra endmembers from improving the score through a larger model.
  • All four metrics are lower-is-better and are averaged across repeated runs to assess complementary aspects of unmixing quality.They cover cardinality recovery, spectral compatibility, abundance fidelity under mismatch, and reconstruction quality.

B. Tool-Call Statistics

Across modular pipelines, the agent performs repeated refinement actions and frequently retrieves spectral-library evidence. HYDICE Urban generates longer trajectories and more state changes than the other evaluated datasets.

  • 18.12 iterations, 5.98 discard operations, and 6.81 merge operations occur on average per run.
  • 7.60 spectral-library accesses occur on average per run across the datasets.Retrieval remains frequent throughout refinement.
  • HYDICE Urban produces substantially longer trajectories and more state changes than Jasper Ridge and Stonewall Playa.This pattern is consistent with its larger initial candidate sets.

C. Additional Results on Samson Dataset

On Samson, agentic refinement substantially improves modular decompositions, especially endmember-count and abundance estimation, while remaining competitive with end-to-end methods. The results also show dependence on initial decomposition quality and incomplete correction of severe model-order errors.

  • Dataset: Samson contains three endmembers—soil, trees, and water—in the commonly used 95×95×156 subscene.The original image contains 952×952 pixels and 156 spectral bands spanning 0.401 to 0.889 µm.
  • Agentic refinement: Under HySime, refinement reduces ∆K from 40 to 8.0–17.4 and improves aRMSE for all four backbones.
  • Agentic refinement: Under NWHFC, refinement reduces ∆K from 5 to approximately one endmember and consistently lowers both aRMSE and rRMSE.
  • Agentic refinement: VCA–UnDIP reaches ∆K = 0.7, aRMSE = 0.304, and rRMSE = 0.041 after refinement.These are the strongest reported Samson results in the modular-refinement comparison.
  • Comparison: Under NWHFC, refined pipelines collectively win all four metrics against dedicated end-to-end methods, whereas CNN-AE remains better in ∆K and aRMSE after severe HySime overestimation.Weaker SISAL results indicate dependence on the quality of initial candidate signatures.

D. Additional Ablations

The ablation examines how merge strategies affect refined unmixing, finding different trade-offs among cardinality, spectral, abundance, and reconstruction accuracy.

  • Three merge strategies are compared: average, weighted average, and medoid.The comparison uses HYDICE Urban with HySime–VCA–UnDIP initialization.
  • Simple averaging achieves the lowest cardinality error.
  • The medoid strategy yields the best abundance and reconstruction accuracy.
  • Weighted averaging obtains the lowest spectral error and ranks second on both RMSE measures, but has a higher ΔK.
  • Weighted averaging is adopted because it prioritizes endmember fidelity.

E. Qualitative Analysis of Recovered Endmembers

The qualitative analysis compares recovered spectra with references and illustrates how the agent combines spectral and spatial evidence to validate, identify, and refine endmembers.

  • Recovered spectral signatures: The agent achieves lower spectral angle distance than CNN-AE for 13 of 16 reference endmembers.It performs better on five of six HYDICE Urban materials, all four Jasper Ridge materials, one of three Stonewall Playa materials, and all three Samson materials.
  • Recovered spectral signatures: CNN-AE performs better only for grass on HYDICE Urban, while the agent is particularly stronger for tree, roof, and metal.
  • Recovered spectral signatures: Stonewall Playa shows mixed results: the agent is better for Montmorillonite/Illite, whereas CNN-AE is better for Alunite and desert varnish.
  • Agent execution: The qualitative section traces one HYDICE Urban execution from initialization through tool calls to the final active-endmember report.
  • Spatial validation: The agent validates candidate materials using abundance-map spatial cohesion and mass gatekeeping, including vegetation and soil, while discarding noise artifacts.
  • Spectral identification: After spatial validation, spectral-library searches ground ambiguous matches by cross-referencing mineral and artificial-surface hypotheses with spatial evidence.
Loading 2609.01289v1…