Source-linked AI summary
SPEAR: Structure Property Explainability with Attention Regularization
Aditya Raghavan, Utkarsh Pratiush, Dalton A. Pearl, Jade Holliman, Katharine Page, Philip D Rack, Sergei V Kalinin
TL;DR
Attention-based models for structure–property prediction can be difficult to interpret when unregularized attention produces unstable attributions. SPEAR regularizes attention during training, yielding physically aligned explanations while preserving regression performance and identifying a relationship involving the {220} reflection.
Problem
Standard, unregularized attention can produce unstable attribution patterns, limiting interpretability in structure–property prediction.
Method
SPEAR is a learnable attention framework that regularizes attention in supervised regression from spectral inputs.
Results
Regularization produced smooth, contiguous attributions aligned with physical features while maintaining equivalent regression performance and highlighting the {220} reflection.
Takeaways & Limitations
Attention regularization provides a physically motivated inductive bias that improves interpretability in attention-based structure–property models.
Abstract
from arXiv · showhide
Machine learning is increasingly used to learn structure property relationships from spectroscopic and diffraction data, yet its adoption in materials discovery is often limited by poor interpretability of model predictions. Although attention mechanisms are frequently treated as inherently explainable, unregularized attention can yield unstable, fragmented, or intensity driven attribution patterns that obscure the physical origin of these relationships. Here we introduce SPEAR (Structure Property Explainability with Attention Regularization), a framework that constrains attention distributions during training to improve their stability, selectivity, and physical interpretability. SPEAR augments attention based regression with a learnable temperature that controls attention concentration and a smoothness penalty that enforces coherence across neighboring spectral positions, treating attention as a learnable explanatory object rather than a post hoc visualization. Using synthetic spectral benchmarks with known generative structure, we show that attention regularization produces smooth, contiguous attribution profiles aligned with causal features while preserving predictive accuracy. Applied to experimental X ray diffraction data from a combinatorial rare earth zirconate thin film library, the regularized model selectively emphasizes physically relevant diffraction features and decouples feature importance from raw peak intensity. The reflection it identified prompted a reassessment of our earlier structural analysis, revealing a correlation between the 220 peak position, the tetragonal distortion that accommodates cation size disorder, and the local thermal conductivity. Attention regularization therefore provides a principled training constraint for explainable structure property regression, yielding mechanistically meaningful explanations without sacrificing predictive performance.
I. Introduction
Materials ML can predict properties from diffraction data, but unreliable explanations may misidentify physically relevant features and misdirect experiments. SPEAR addresses this interpretability gap by training attention to produce coherent, physically meaningful attributions.
- Motivation and challenge: Accurate material-property prediction does not guarantee physical understanding, because explanations may track irrelevant intensity or the wrong data region.Such explanations can lead researchers toward incorrect compositions, synthesis conditions, or material descriptors.
- Motivation and challenge: Correlated peak shifts in combinatorial XRD libraries make it difficult to distinguish the peak that drives a property change from peaks that merely co-vary with it.Because explanations guide subsequent experiments and theory development, diffuse or misleading attribution can actively misdirect scientific decisions.
- High-throughput XRD: Manual structural refinement does not scale to compositionally complex, disordered libraries containing hundreds or thousands of diffraction patterns, shifting the goal toward library-wide composition–structure–property trends.The passages identify machine learning as suitable for processing large pattern sets without a pre-specified structural model, while retaining an interpretability gap.
- Limits of post hoc attribution: SHAP can diffuse importance across correlated spectral regions and is applied after training, so its explanations are not guaranteed to be stable or physically consistent.This can assign high importance to multiple co-varying peaks even when only one is physically responsible for the target property.
- SPEAR framework: SPEAR introduces a framework that constrains attention during learning so the resulting weights are spatially coherent and physically meaningful from the start.The framework is evaluated on a diffraction peak-position task and a distributed structural-contribution task predicting thermal conductivity, showing that constrained attention can reliably identify meaningful diffraction features.
II. Method Description:
The method uses attention as a structured pooling mechanism for spectral regression, aggregating encoded spectral features before regression while producing interpretable weighting over spectral regions. These weights indicate predictive sensitivity and relative contribution, rather than causal effects, and require careful interpretation or constraints.
- Interpretation: Attention weights do not universally guarantee causal explanations, but careful interpretation or appropriate constraints can make them meaningful for generating hypotheses about model behavior.The method treats attention as an interpretability signal while acknowledging its limitations.
- Interpretation: Attention-based attribution is interpreted as predictive sensitivity rather than causal effect.Unlike causal inference methods, this approach does not explicitly reconstruct causal relationships.
- Method workflow: Attention weights aggregate encoded spectral features before regression, producing an interpretable weighting over spectral regions.Spectra are discretized into positionally encoded tokens, mapped to latent representations by an encoder, and pooled with learned attention weights.
- Method workflow: The model selectively emphasizes informative spectral regions while reflecting their relative contribution through the attention distribution.Attention-weighted aggregation is performed before regression, allowing spectral regions to receive different learned weights.
II A. Experimental Dataset and Preprocessing
The study uses synchrotron X-ray diffraction spectra from a combinatorial rare-earth zirconate thin-film library to model structural and thermal-property variation. Spectra undergo normalization, compression, downsampling, and coordinate encoding before regression of a known peak position and thermal conductivity.
- Experimental dataset: Synchrotron X-ray diffraction measurements sampled a (Gd,Dy,Ho,Er)2Zr2O7 thin-film library with broad disordered defect-fluorite features and composition-dependent peak shifts and prominence changes.The observed variations are consistent with spatially varying lattice spacing and disorder-related structural changes.
- Preprocessing: Each wafer location was treated as a composition, and its processed one-dimensional XRD spectrum was used as the model input.Spectra were matched to metadata by measurement point ID, with wafer coordinates converted from millimeters to centimeters.
- Preprocessing: Each spectral token retained processed intensity and normalized 2θ coordinate, preserving the full diffraction profile rather than fitted descriptors.Before transformer training, intensities were shifted non-negative, scaled by each spectrum’s 95th percentile, and transformed with log(1 + I) compression.
II B. Problem Setup and Model Overview
The problem is formulated as supervised scalar-property regression from XRD spectra represented as ordered sequences. A transformer regressor uses self-attention and attention-based pooling to aggregate spectral information, with synthetic targets defined by embedded Gaussian-peak amplitudes.
- The model predicts a scalar target property from XRD spectral inputs.
- Ordered spectral measurements are treated as sequences to preserve continuity among neighboring diffraction features.
- A transformer-based regressor maps each input spectrum to a scalar output using contextualized representations from self-attention layers.
- In synthetic benchmarks, the regression target is the amplitude of a Gaussian peak embedded within the spectrum.
- Attention-based pooling assigns normalized weights across spectral positions and forms a weighted sum of their hidden states.
II C. Plain vs. Learnable Attention Pooling · II D. Attention Formulation
Plain attention uses softmax-normalized weights optimized only through prediction loss, leaving their sharpness, locality, and stability unconstrained. SPEAR instead regularizes attention during training as a learnable explanatory object, while retaining the same backbone for controlled interpretability comparisons.
- II C. Plain vs. Learnable Attention Pooling: Plain attention obtains weights through a softmax over attention logits and optimizes them solely through prediction loss.
- II C. Plain vs. Learnable Attention Pooling: Its sharpness, locality, and stability emerge implicitly from optimization rather than explicit control.
- II C. Plain vs. Learnable Attention Pooling: SPEAR treats attention as a learnable explanatory object whose distributional properties are shaped during training.
- II C. Plain vs. Learnable Attention Pooling: Explicit regularization controls attention concentration and smoothness, enabling selective attendance while enforcing physically motivated constraints.
- II C. Plain vs. Learnable Attention Pooling: The resulting regularized attention distributions are more stable and interpretable.
- II C. Plain vs. Learnable Attention Pooling: Plain and regularized models share the same backbone architecture and differ only in attention regularization terms, enabling controlled interpretability comparisons.
- II D. Attention Formulation: The transformer encoder formalizes attention through scaled dot-product self-attention using contextualized representations.
- II D. Attention Formulation: Softmax normalization makes attention weights a probability distribution over key positions that quantifies relative importance across spectral positions.
II E. Loss Terms
SPEAR augments prediction loss with explicit attention regularization to control concentration and smoothness during training. The resulting objective jointly promotes predictive and interpretable attention, while disabling regularization recovers plain attention.
- Motivation: Prediction loss alone can produce unstable, excessively sharp, overly uniform, or high-frequency attention patterns that undermine interpretability.These behaviors may not strongly affect predictive accuracy but weaken attention as meaningful importance weighting.
- Prediction loss: The prediction component uses mean squared error to learn the target property.In the synthetic setting, the target property is the Gaussian peak amplitude.
- Attention concentration control: A learnable temperature τ rescales attention logits, allowing the model to adaptively control concentrated or diffuse weighting.Smaller τ promotes more concentrated attention, whereas larger τ yields more diffuse weighting; a weak quadratic prior prevents degenerate extremes.
- Attention smoothness: A discrete smoothness penalty discourages implausible high-frequency fluctuations across neighboring spectral positions.The penalty is averaged across batches, attention heads, and query positions and scaled by λsmooth.
- Combined objective: Together, the loss terms constrain attention during training to produce distributions that are both predictive and interpretable.The regularized objective controls attention concentration and coherence across neighboring positions.
- Plain attention: With regularization disabled, the formulation reduces exactly to standard attention pooling optimized solely through prediction loss, termed plain attention.This defines the limiting-case baseline used throughout the paper.
II F. Training algorithm
SPEAR trains the attention-based regressor end-to-end with stochastic gradient optimization, combining prediction loss with attention concentration and smoothness regularization. A learnable temperature is optimized by backpropagation, while disabling regularization recovers standard attention pooling.
- Training algorithm: Input spectra are tokenized and encoded into latent feature representations, from which attention logits are computed.The complete procedure is summarized in Table 1.
- Training algorithm: A temperature-scaled softmax produces the attention distribution, whose weighted pooling forms a global representation for property regression.The pooled representation is passed to the regressor to predict the target property.
- Training algorithm: The training objective combines prediction loss with attention concentration and smoothness regularization terms.These terms are optimized jointly with the predictive objective.
- Training algorithm: Gradients of the combined loss update all model parameters, including the attention temperature τ, through backpropagation and Adam optimization.The attention temperature is therefore learned during training rather than fixed separately.
- Training algorithm: When attention regularization is disabled, the procedure reduces exactly to standard attention pooling optimized solely with the prediction objective.This provides the unregularized training configuration described by the method.
III. Results · III A. Synthetic data
On synthetic spectra with a known target, SPEAR produces smooth, localized attention aligned with the causal peak, unlike fragmented intensity-driven plain attention. Moderate regularization improves interpretability while preserving or minimizing prediction error, whereas insufficient or excessive regularization harms attribution quality or performance.
- III A. Synthetic data: The benchmark used 500 noisy spectra containing three Gaussian peaks, with the amplitude of the third peak as the regression target.Peak amplitudes, positions, widths, and Gaussian noise were randomly sampled to mimic experimental conditions.
- III A. Synthetic data: Plain attention spread across all three prominent peaks, tracking peak prominence rather than isolating the supervised target.SHAP attribution for the same model concentrated almost exclusively on the third peak, exposing attention’s unfaithful predictive sensitivity.
- III A. Synthetic data: SPEAR generated smooth, contiguous attention spanning the target peak while suppressing spurious emphasis on neighboring peaks.Agreement between attention attribution and SHAP indicated that the regularized distribution reflected predictive behavior rather than an arbitrary pattern.
- III A. Synthetic data: Both models accurately recovered target amplitudes, with nearly identical final validation errors and slightly smoother, more stable early convergence for SPEAR.Their validation parity plots closely followed the ideal diagonal, and both models converged rapidly during training.
- III A. Synthetic data: A parameter sweep revealed an intermediate regularization regime with minimized validation error, while weak or absent regularization produced unstable attributions and excessive regularization degraded performance.The results establish a controllable balance between predictive accuracy and interpretability.
- III A. Synthetic data: Unregularized attention was fragmented and sharply fluctuating across the peak, whereas balanced regularization yielded smooth, localized attention aligned with the lowest validation error.Balanced constraints spanned the physically relevant diffraction peak and suppressed contributions elsewhere in the spectrum.
- III A. Synthetic data: Over-smoothing broadened attributions beyond the peak, diluted selectivity, and increased prediction error.Thus, smoothness must be tuned to the physical scale of meaningful features rather than imposed indiscriminately.
III B. Experimental data – Peak Feature
Experimental XRD patterns share a single defect-fluorite reflection set but vary systematically in peak positions and relative intensities across the wafer. SPEAR selectively attributes the target peak while preserving accurate predictions and accelerating convergence, unlike plain attention’s intensity-driven attributions.
- III B. Experimental data – Peak Feature: The library retains a single defect-fluorite phase, while peak positions and relative intensities vary systematically across wafer locations.These variations, rather than reflection appearance or disappearance, provide the extracted descriptors.
- III B. Experimental data – Peak Feature: Diffraction-derived descriptor maps show smooth, reproducible gradients across the library, consistent with underlying compositional variation.Descriptors include the first three peak positions and the extracted lattice parameter.
- III B. Experimental data – Peak Feature: Plain attention and SHAP distribute attribution across prominent reflections, closely tracking peak intensity rather than isolating the causally relevant feature.The model effectively soft-reweights the full spectrum, emphasizing stronger reflections regardless of target relevance.
- III B. Experimental data – Peak Feature: SPEAR concentrates attention and SHAP on the second diffraction peak defining the target, suppressing other peaks even when their intensities are comparable or larger.Its attribution is localized and smoothly distributed across the relevant peak’s width, decoupling importance from raw peak magnitude.
- III B. Experimental data – Peak Feature: Both models predict accurately, but regularized attention converges more rapidly through a faster early-epoch reduction in validation loss.Parity plots closely follow the ideal diagonal, while SPEAR’s selective attention simplifies identification of the underlying structure–property relationship.
III C. Experimental data – Material Property
On experimental XRD data, SPEAR concentrates attention on physically relevant diffraction features while preserving accurate thermal-conductivity prediction. Its identified {220} reflection links peak-position, width, and asymmetry variations to tetragonal distortion and thermal conductivity.
- Experimental XRD attribution: Plain attention and SHAP distribute importance across strong diffraction peaks, largely following intensity and offering limited physical guidance.The broad attribution reflects the correlated nature of the diffraction pattern rather than isolating specific predictive regions.
- Experimental XRD attribution: SPEAR and SHAP concentrate on a limited subset of diffraction features, including lower-intensity peaks, with agreement indicating genuine predictive sensitivity.The emphasized regions correspond to planes sensitive to lattice disorder and bonding environments associated with phonon scattering.
- Predictive performance: Both models predict thermal conductivity accurately, while SPEAR shows reduced scatter, faster convergence, and slightly lower validation loss.These outcomes are consistent with focus on a smaller, more informative set of spectral regions.
- Structural interpretation: The {220} peak position, width, and asymmetry vary systematically with tetragonal distortion across the library, connecting the SPEAR attribution to thermal conductivity.The model therefore generated a physically grounded, testable structural hypothesis rather than merely highlighting prominent intensity features.
Conclusion
SPEAR regularizes attention during training to produce stable, selective, physically interpretable explanations without sacrificing regression performance. In synthetic and experimental diffraction data, it aligned attribution with causal structure and generated a previously unidentified structure–property correlation.
- Motivation: Unregularized attention produced fragmented, intensity-driven, unstable patterns that obscured underlying physics in spectroscopic and diffraction data.These patterns were difficult to interpret physically.
- Method: SPEAR introduced explicit regularization controlling attention concentration and smoothness, embedding interpretability constraints directly into learning rather than applying them post hoc.The framework treats attention as a learnable explanatory object.
- Synthetic benchmarks: Synthetic benchmarks showed that regularization yielded smooth, contiguous attribution profiles aligned with physical features while maintaining equivalent regression accuracy.Ablations identified an intermediate regime that improved interpretability and predictive stability; insufficient or excessive regularization degraded attribution quality and/or performance.
- Experimental XRD application: On the (GdDyHoEr)2Zr2O7 thin-film library, SPEAR preferentially emphasized target-relevant diffraction peaks instead of broadly weighting high-intensity features, decoupling relevance from raw intensity.Its improved attribution accompanied comparable or modestly improved convergence behavior and prediction error.