Source-linked AI summary
DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction
Weiran Wang, Xintong Huo, Yueying Wang, Yusi Fan, Wenyan Wang, Xin Feng, Ruihao Xin, Lan Huang, Kewei Li, Fengfeng Zhou
TL;DR
Low-data materials property prediction is challenged by scarce labels and the structural-input requirements of accurate models. DISTAL combines self-supervised compositional pretraining with ALIGNN-based structural distillation, then integrates the resulting signals for composition-only inference. Its best multimodal configuration outperforms the reference benchmark on 37 of 39 tasks.
Problem
Materials property prediction is difficult when labeled samples are scarce, while strong structure-aware models require crystal structures that may be unavailable during early screening.
Method
DISTAL combines self-supervised compositional pretraining with structural knowledge distillation from a pretrained ALIGNN teacher into a composition-conditioned student.
Results
The best multimodal configuration outperformed the reference benchmark on 37 of 39 tasks and achieved the lowest benchmark-wide mean.
Takeaways & Limitations
Compositional pretraining and structural distillation provide complementary priors for composition-only prediction in small-data materials informatics.
Takeaways & Limitations
Some properties may remain difficult to capture from composition-only inputs.
Abstract
from arXiv · showhide
Materials property prediction remains difficult in low-data settings, where many target properties are supported by only a limited number of labeled samples. Models with the strongest predictive accuracy often depend on crystal structures, which restricts their use in early-stage screening when structural information is limited or unavailable. To address this challenge, we propose DISTAL, a dual-prior framework for structure-agnostic materials property prediction that combines self-supervised compositional pretraining with structure-aware knowledge distillation. DISTAL first learns transferable compositional representations from a large virtual composition space using 145 composition-derived descriptors. It then distills structural knowledge from a pretrained ALIGNN teacher into a composition-conditioned student. This setting allows structural priors to be used during training without requiring structural inputs at inference. By integrating explicit compositional descriptors, pretrained latent features, and distilled structural features within a unified prediction pipeline, DISTAL captures complementary signals that are difficult to recover from any single representation alone. Across 39 benchmark tasks, the best-performing multimodal configuration combines all three signals, and improves over the reference benchmark on 37 tasks. DISTAL achieves the strongest overall performance among all evaluated feature combinations. These results indicate that compositional pretraining and structural distillation provide complementary priors and offer a practical route to robust composition-only prediction in small-data materials informatics. The source code and the pre-trained models are anonymously available at: https://osf.io/eq96d/overview?view_only=451617f42f7849e08750bd1852b48980 and will be released at the official link after acceptance.
Introduction
Materials property prediction faces a tension between structure-aware accuracy and composition-only deployability, compounded by scarce labels. DISTAL addresses this gap by combining compositional pretraining with structural distillation for composition-only inference.
- Structure-aware models provide strong geometric priors but require crystal structures that may be unavailable, uncertain, or expensive during early screening.
- Composition-only models improve scalability and avoid the structural bottleneck, but they do not directly observe geometry governing phase-, coordination-, or polymorph-dependent behavior.
- Many materials properties have too few labeled samples to support stable training of expressive models from scratch.
- Transfer learning and self-supervised pretraining offer low-data benefits, but these approaches have often developed along separate directions.
- DISTAL combines self-supervised compositional pretraining and ALIGNN-based structural knowledge distillation within a single composition-only inference pipeline.
- Across 39 benchmark tasks, the study evaluates explicit descriptors, pretrained compositional features, distilled structural features, and their combinations as complementary signals.
Materials and Methods
DISTAL uses shared composition descriptors to learn compositional representations and distill structural information into a student model. Downstream evaluation combines these feature types while requiring only composition-derived inputs for inference.
- DISTAL uses a shared 145-dimensional composition-derived descriptor vector throughout pretraining, distillation, and downstream evaluation.
- Self-Supervised Compositional Pretraining: Self-supervised pretraining maps composition descriptors to latent features and reconstructs the input descriptor vector through a decoder.
- Structural Distillation: Structural distillation transfers information from a pretrained ALIGNN teacher into a composition-conditioned student representation.
- Structural Distillation: The student receives composition-derived descriptors, while structural information is transferred during training and omitted from downstream inference.
- Downstream Evaluation: Downstream experiments evaluate explicit descriptors, latent compositional features, distilled structural features, and their combinations across 39 property-prediction tasks.
- Pretraining Corpus: The virtual pretraining corpus contains 100,000 randomly sampled compositions and expands compositional coverage without defining a chemically curated database.
Results and Discussion
Across 39 tasks, raw descriptors, self-supervised compositional features, and distilled structural features each contributed useful but non-identical information. Their fusion produced the strongest overall composition-only performance, improving over the reference benchmark on 37 tasks.
- Self-supervised compositional pretraining: Self-supervised pretraining improved downstream prediction across many tasks, but the useful feature type depended on the autoencoder backbone and branch.The latent and reconstruction branches captured different views of the learned representation, so no candidate was uniformly optimal for both.
- Structural knowledge distillation: The distilled structural branch was the strongest standalone feature type, improving over the reference benchmark on 33 of 39 tasks.This indicates that structure-aware supervision remained informative after transfer into a composition-conditioned representation that requires no crystal structures at inference.
- Feature fusion: The three-way fusion of latent, explicit, and distilled features achieved the strongest overall result, improving on 37 of 39 tasks and attaining a mean test MAE of 39.89.The result supports complementarity: explicit descriptors, structural distillation, and latent compositional features contributed distinct signals.
- Task-wise comparison: The final DISTAL model improved across diverse property families, although magmom_outcar and piezo_max_dij remained below the reference benchmark.These exceptions indicate that some targets remain difficult to capture from composition-only inputs even after pretraining and structural distillation.
Conclusions
DISTAL combines compositional pretraining and distilled structural information within a composition-only inference pipeline for low-data materials property prediction. Across 39 tasks, integrating explicit descriptors, learned compositional features, and distilled structural features produced the strongest overall performance.
- Conclusions: DISTAL combines self-supervised compositional pretraining with structure-aware knowledge distillation in a unified composition-only prediction framework.The framework retains composition-only inference while incorporating transferable compositional regularities and distilled structural information.
- Conclusions: 145-dimensional composition-derived descriptors provide a strong baseline for structure-agnostic materials property prediction.The descriptors are derived from a large virtual composition space.
- Conclusions: Self-supervised compositional pretraining improves downstream transfer in a representation-dependent manner, while ALIGNN-guided distillation produces the strongest standalone learned branch.The conclusions describe these effects across the evaluated representation and feature types.
- Conclusions: The multimodal analysis finds that explicit descriptors, learned compositional features, and distilled structural features are complementary rather than redundant.The analysis motivates combining their distinct strengths rather than relying exclusively on handcrafted descriptors, compositional learning, or transferred structural priors.
- Conclusions: 37 of 39 tasks improved over the reference benchmark, and the final z + x + s configuration achieved the lowest benchmark-wide mean MAE among evaluated feature modes.This result was obtained under the adopted evaluation protocol.
- Conclusions: DISTAL is a practical route for transferring part of structural supervision into settings where only composition is available at inference, including early-stage screening.The paper does not present DISTAL as a replacement for direct structure-aware modeling, and notes future extensions involving constrained virtual compositions, richer teachers, and task-aware workflows.