PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-Spectrum: Multimodal Multi-spectral Molecular Structural Elucidation with a Stable MoE Framework

Hai-tao Yu, Nan Min, Zheng Fang, Hongyu Zhan, Yusen Tan, Yuhan Wang, Jun Xia

arXiv:2608.27286v1cs.LG

TL;DR

Multispectral sequence concatenation can degrade molecular structure elucidation because heterogeneous modalities create imbalance. MM-Spectrum addresses this with modality-aware routing, structured experts, heterogeneous capacities, and cost regularization, achieving consistent improvements across full-modality, bimodal, and missing-modality settings.

  • Problem

    Direct multispectral concatenation can substantially degrade molecular structure elucidation because heterogeneous modalities create imbalance and allow weak modalities to dominate attention.

  • Method

    MM-Spectrum uses modality-aware routing, shared and interaction experts, heterogeneous expert capacities, and computation-cost regularization to separate modality-specific, redundant, and synergistic information.

  • Results

    MM-Spectrum achieves consistent improvements across full-modality, bimodal, and missing-modality settings, supported by ablation and interpretability analyses.

  • Takeaways & Limitations

    MM-Spectrum provides a structured approach for mitigating multispectral imbalance and extracting cross-spectral synergy, while yielding substantial gains in the full-modality regime.

  • Takeaways & Limitations

    The framework uses Partial Information Decomposition as a theoretical inductive bias, positing that multimodal information decomposes into redundant, unique, and synergistic components.

Abstract

from arXiv · show

Inferring molecular structures from multimodal spectroscopic measurements requires integrating complementary yet highly heterogeneous signals. However, the common paradigm of directly concatenating multispectral sequences can exhibit anomalous performance degradation, primarily due to pronounced heterogeneity and the resulting multimodal imbalance across modalities. As a remedy, we propose MM-Spectrum, a sparse Mixture-of-Experts framework tailored for multimodal multispectral spectra-to-structure elucidation. To better match the information characteristics under multispectral imbalance, MM-Spectrum introduces an explicit modality-aware routing mechanism that exposes spectral identity to the router in addition to token content representations. Moreover, it incorporates shared and interaction experts, together with heterogeneous expert capacities, to extract multispectral modality-unique and cross-modal synergistic information while suppressing noise-induced interference. Across full-modality, bimodal, and missing-modality settings on molecular structural elucidation, MM-Spectrum achieves consistent and substantial improvements, supported by ablation studies and interpretability analyses.

1. Introduction

Multispectral molecular elucidation is hindered by heterogeneous modalities and imbalance, which can make naive full-modality fusion underperform. MM-Spectrum replaces simple concatenation with modality-aware sparse experts that separate modality-specific, redundant, and synergistic information, improving performance across settings.

  • Motivation: NMR provides high-resolution local-topology constraints, whereas IR offers noisier, lower-density functional-group evidence requiring longer fragments.
  • Motivation: Multispectral fusion can degrade when long, weak modalities dominate attention, causing full-modality models to underperform unimodal or bimodal baselines.The paper attributes this failure primarily to drastic modality-distribution differences and multimodal imbalance.
  • Method: MM-Spectrum uses modality-aware routing that exposes spectral identity alongside token content to allocate experts under multispectral imbalance.The framework reconstructs fusion as structured expert division of labor rather than crude additive aggregation.
  • Method: Shared and interaction experts capture cross-modality redundancy and synergy, while heterogeneous capacities and computation-cost regularization suppress noise-induced interference.The design targets modality-unique and synergistic information pathways.
  • Contribution: The paper identifies multimodal imbalance induced by heterogeneity and interprets it through information-density disparity and gradient conflicts.
  • Results: MM-Spectrum demonstrates consistent improvements in full-modality, bimodal, and missing-modality settings, supported by ablations and interpretability analyses.

2. Related Work

Molecular structural elucidation has progressed from database-driven retrieval and classical construction methods to deep generative models for individual spectroscopic modalities. Multimodal systems still commonly rely on naive early fusion, while MM-Spectrum adapts sparse MoE principles to spectral heterogeneity and imbalance.

  • Prior approaches: Classical elucidation methods retrieve candidates from molecular databases using spectral fingerprints or construct structures from NMR information.
  • Deep generative models: Deep generative approaches include autoregressive and diffusion models for mass spectra, token-based and graph-based methods for NMR, and contrastive learning for IR.
  • Multimodal learning: In multimodal elucidation, the prevailing baseline remains naive early fusion, concatenating or pooling heterogeneous modalities before processing.
  • Mixture-of-Experts: Sparse MoE models increase capacity by conditionally activating a subset of parameters through learned top-k routing, while auxiliary balancing losses address expert collapse.
  • Mixture-of-Experts: Standard uniform balancing assumes relatively homogeneous token information and can hinder specialization by assigning high-capacity experts to low-utility noise tokens.MM-Spectrum instead uses modality-aware routing and cost regularization as spectrum-specific inductive biases.

3. Methodology

MM-Spectrum reformulates multimodal spectra-to-structure generation as a structured sparse-MoE encoder–decoder that explicitly accounts for modality imbalance and decomposes information by function. It combines modality-aware routing, modality-specific compression, PID-inspired expert partitions, and heterogeneous expert capacities.

  • Task Definition: MM-Spectrum formulates molecular structural elucidation as an encoder–decoder task that generates SMILES sequences from multimodal spectra.The encoder produces contextual representations and the decoder generates the target structure autoregressively.
  • Motivation: Standard content-only MoE routing can neglect sparse, constraint-rich NMR signals when abundant low-density IR or MS tokens dominate routing statistics.This sequence imbalance motivates modality-aware routing and spectrum-specific computation allocation.
  • Spectrum-Aware Representation: Modality-specific operators compress redundant IR and MS signals while approximately preserving information-rich NMR topology to align effective token densities.Local binning is used for autocorrelated IR, whereas top-k filtering removes noisy MS background.
  • Modality-Aware Routing: The router combines token content with learnable modality tags and biases before selecting experts through modality-conditioned probability distributions.This supports soft specialization without hard-coding experts to individual modalities.
  • Structured Expert Space: PID-inspired expert partitions assign shared experts to redundancy, modality-specific experts to unique constraints, and interaction experts to cross-modal synergy.Consistency and separability regularizers encourage invariant shared representations and distinct modality-specific features.
  • Heterogeneous Experts: Heterogeneous Heavy and Light experts, combined with computation-cost regularization, allocate expensive processing when task-loss reduction justifies the cost.The objective promotes Heavy-for-hard and Light-for-easy specialization while targeting high-utility tokens.

4. Experiments

Experiments evaluate MM-Spectrum across modality combinations, missing-modality conditions, molecular complexity, real experimental spectra, and interpretability-oriented ablations. Across these settings, MM-Spectrum consistently improves robustness and accuracy over dense concatenation while supporting specialized routing and capacity allocation.

  • Benchmark and Evaluation: The benchmark pairs molecular structures with NMR, IR, and MS measurements, using Top-K SMILES accuracy and identical decoder and non-MoE components for comparison.The implementation uses a Transformer encoder–decoder backbone with eight MoE experts, Top-k = 2 gating, and beam width 10.
  • Modality Combination Study: 44.29% marks the Dense baseline’s full-modality accuracy, significantly below the unimodal NMR baseline.The result is interpreted as catastrophic negative transfer from long, low-density IR/MS inputs overwhelming optimization and diluting high-fidelity signals.
  • Modality Combination Study: 76.04% is MM-Spectrum’s full-modality accuracy, reversing the Dense baseline’s 44.29% collapse and yielding consistent Pareto improvements across settings.The structured expert allocation selectively integrates complementary evidence rather than relying on naive concatenation.
  • Missing-Modality Robustness: MM-Spectrum shows smoother degradation and superior Top-10 candidate recoverability across seven test-time missing-modality settings.The baseline instead shows sharp performance drops as missingness increases, without test-time adaptation.
  • Generalization to Experimental Spectra: 14.10% versus 18.15% Top-1 accuracy shows Dense concatenation degrading on full-modality SDBS inputs relative to standalone NMR.The comparison is reported for models trained from scratch on empirical laboratory measurements.
  • Complexity-Stratified Evaluation: MM-Spectrum improves across all HAC strata, with the largest relative gains on medium- and large-complexity molecules.The reported trend is linked to high-capacity and interaction-specific experts being activated for difficult instances.
  • Interpretability and Training Dynamics: MM-Spectrum develops stable, well-separated representations while routing evolves from diffuse exploration to persistent specialized hotspots across layers.Expert utilization varies by depth without collapse or uniformity, indicating distinct layer-wise functional roles.
  • Ablation Studies: Removing Router-aware signals or Tag routing consistently reduces Top-K accuracy, with the largest relative degradation at Top-1.The ablation supports explicit modality cues for candidate coverage and ranking precision; heterogeneous capacities and computation-aware regularization likewise provide combined benefits.

5. Conclusion

MM-Spectrum is presented as a sparse MoE framework for optimization instability in multimodal molecular structural elucidation caused by severe spectral heterogeneity. It improves the full-modality regime by combining modality-aware routing, structured expert subspaces, and heterogeneous computation, while not targeting specialized single-modality models.

  • Conclusion: MM-Spectrum combines spectroscopy-aware compression, explicit modality routing, structured expert subspaces, and curriculum-driven heterogeneous computation.The design is intended to mitigate imbalance-induced conflicts and extract cross-spectral synergy.
  • Conclusion: The framework resolves full-modality collapse and yields substantial gains in the full-modality regime.The conclusion distinguishes this goal from outcompeting highly specialized single-modality models.

Impact Statement

The work aims to advance multimodal machine learning for chemical structure elucidation and may support drug discovery and materials science research. The authors report no anticipated negative ethical or adverse societal consequences.

  • Impact Statement: The stated goal is to advance multimodal machine learning for scientific discovery, specifically chemical structure elucidation.The impact statement identifies potential applications in drug discovery and materials science research.
  • Impact Statement: The authors do not foresee negative ethical impacts or adverse societal consequences from this work.

A.1. Notation and Task Objective

The paper frames multispectral elucidation as a multi-objective problem in which token-count and information-density imbalance can create gradient conflict. MM-Spectrum addresses this with modality-aware routing and structured, cost-aware expert allocation.

  • Multispectral inputs are represented as modality-specific token sequences mapped to a target SMILES sequence by an encoder–decoder model.
  • Negative transfer occurs when cross-modal gradients conflict, with cos(g_i, g_j) < 0 indicating that improving one modality tends to degrade another.
  • Early fusion can let long, low-density modalities dominate shared attention and gradient updates, diluting high-density constraints and worsening conflicts.
  • MM-Spectrum injects modality tags and biases into routing so expert preferences reflect spectral identity while retaining content-based specialization.
  • Its training curriculum moves from broad expert coverage to alignment and then cost-effective specialization by annealing balancing regularization and router temperature.
  • PID-inspired expert families separate shared redundancy, modality-specific constraints, and cross-modal synergy, while heterogeneous capacities and cost penalties allocate compute selectively.

B.5. Data Preprocessing Details

The benchmark uses modality-specific preprocessing for NMR, IR, and MS before model evaluation. Additional experiments compare fusion strategies under the same general evaluation framework.

  • NMR is tokenized into 0.1 ppm chemical-shift bins and multiplicities, with sequences truncated to 256 tokens.
  • IR is binned into 1024 frequency intervals and its intensities are normalized to [0, 1].
  • MS preprocessing retains the top 100 peaks according to relative intensity.
  • The study also benchmarks Cross-Attention Fusion, Contrastive Alignment, PCGrad, Gradient Blending, and OGM-GE against MM-Spectrum.
  • +31.75% Top-1% accuracy is reported for MM-Spectrum over Dense Concatenation on the canonical full tri-modality benchmark.

C.3. Large-Scale Generalization on the MMST Dataset

On the large-scale simulated MMST dataset, the dense early-fusion baseline deteriorates when all modalities are combined, whereas MM-Spectrum retains stronger full-modality performance.

  • The MMST corpus contains approximately 5,997,971 multispectral samples, including 5,275,360 training, 659,420 validation, and 63,191 test examples.
  • The dense baseline reaches 65.71% full-modality accuracy, below its standalone NMR result of 79.41%.
  • MM-Spectrum reaches 87.35% Top-1 accuracy, a +21.64% margin over the dense multimodal equivalent.

C.4. Robustness to the Complete Absence of an Entire Modality

The missing-modality evaluation tests models trained on complete spectra when one physical channel is entirely removed at inference. MM-Spectrum is reported as more robust than dense early fusion under this setting.

  • At test time, one complete modality channel is zeroed out while models remain trained on full tri-modality spectra.
  • With NMR missing, the Dense baseline falls to 18.65% Top-1 accuracy, whereas MM-Spectrum reaches 39.24%.
  • MM-Spectrum remains above 73% accuracy when either MS or IR is missing.
  • Expert-pool sensitivity favors a moderate pool, with performance saturating or slightly degrading as E becomes too large because of routing dilution.
  • With E = 8 fixed, Top-k = 2 provides the best trade-off; larger k increases compute and routing congestion without comparable accuracy gains.

C.5. Sensitivity Analysis on Expert Capacity and Top-k Routing

The sensitivity analysis identifies how expert-pool size and Top-k routing affect conditional capacity, accuracy, and efficiency. MM-Spectrum’s stable operating point is E = 8 with Top-k = 2, balancing specialization, compute, and routing stability.

  • Expert pool size: Increasing E from 2 or 4 to the default E = 8 consistently improves Top-K performance, while E = 16 offers diminishing returns and may slightly degrade accuracy.The results indicate that richer conditional capacity helps until a saturation regime is reached.
  • Top-k routing: Top-k = 1 reduces computation but limits combining shared, specific, and interaction experts for ambiguous spectral fragments.Allowing limited expert composition generally improves performance over single-expert routing.
  • Operating point: The combined sensitivity results identify E = 8, Top-k = 2 as a stable capacity–utility operating point under modality imbalance.This point provides sufficient specialization without unnecessary compute or routing instability.
  • Motivation: The dense concatenation baseline processes the full multimodal sequence uniformly, incurring quadratic attention cost and spending capacity on low-utility or noisy tokens.This inefficiency is amplified when spectral sequences are long and modality utility is imbalanced.
  • Conditional computation: Sparse MoE routing processes each token with at most k experts and uses cost-aware regularization to reserve heavier computation for tokens with higher marginal utility.The mechanism tends to route low-density or noisy tokens to light experts and difficult tokens to heavy experts.
  • Accuracy and resources: MM-Spectrum outperforms Dense concatenation in both Top-1 and Top-10 accuracy while improving recoverability of correct structures within candidate sets.The comparison uses fixed evaluation and decoding settings, with task performance reported alongside resource measurements.

E. Efficiency Analysis and Heterogeneous Experts

MM-Spectrum uses heterogeneous experts and computation-aware routing to allocate different computational capacities across spectral tokens. Its routing evolves toward stable specialization, while inference uses approximately 40% of the FLOPs of an equivalent-scale dense model and improves accuracy.

  • Heterogeneous experts: MM-Spectrum uses Heavy experts with dffn = 2048 and Light experts with dffn = 512 to match computation with spectral-token complexity.The design assigns more capacity to complex NMR coupling patterns than to simpler IR background noise.
  • Training procedure: The training procedure combines modality-specific preprocessing, explicit modality-aware routing, Top-k expert selection, and task, cost, balancing, entropy, and structural losses.The total objective is optimized with curriculum-weighted auxiliary losses and AdamW.
  • Routing specialization: Heavy Expert activation concentrates on NMR tokens and IR fingerprint regions, whereas Light Experts process low-intensity MS peaks and smooth IR baselines late in training.These patterns show distinct spectral regions acquiring stable computational roles.
  • Efficiency: MM-Spectrum achieves approximately 40% of the inference FLOPs of an equivalent-parameter dense model while reducing latency and improving accuracy.Sparse activation and computation-aware regularization control the active computational footprint despite a larger expert pool.
  • Routing dynamics: The routing dashboard shows an initial coverage phase followed by non-zero stabilized imbalance, indicating movement from random exploration toward structured specialization.The observed transition is consistent with non-collapsed expert allocation.
  • Routing dynamics: Expert-usage bands become stable over time, with different experts capturing persistent roles such as Heavy versus Light processing.The composition plot supports specialization emerging during training.
  • Routing dynamics: Entropy and imbalance exhibit a negative correlation of ρ = −0.889 as training progresses, reflecting a shift from high-entropy exploration to lower-entropy specialized allocation.The trajectory is described as consistent with Pareto optimization.
Loading 2608.27286v1…