Source-linked AI summary
Transfer learning for solvation free energies: from quantum chemistry to experiments
Florence H. Vermeire, William H. Green
TL;DR
Chemical and materials applications face scarce, biased, and noisy experimental data. This paper transfers knowledge from quantum-chemistry calculations into experimental solvation-free-energy models, achieving improved performance for small datasets and out-of-sample cases. The method reaches lower error than the quantum calculations, while experimental noise limits overall performance.
Problem
Deep-learning applications in chemical and materials science face data scarcity, bias, limited coverage, and experimental noise.
Method
The paper transfers parameters from quantum-chemistry models to models fine-tuned on experimental solvation free-energy data using a D-MPNN-based architecture.
Results
Less than ∼0.1 kcal/mol error is achieved for in-scope molecules, with quantum pre-training improving predictions outside the experimental training range and for all molecules.
Takeaways & Limitations
Quantum-chemistry pre-training improves solvation-free-energy predictions for small experimental datasets and out-of-sample solvents, elements, and higher-molar-mass solutes.
Takeaways & Limitations
Performance is limited by aleatoric uncertainty, namely noise in the experimental dataset.
Abstract
from arXiv · showhide
Data scarcity, bias, and experimental noise are all frequently encountered problems in the application of deep learning to chemical and material science disciplines. Transfer learning has proven effective in compensating for the lack in data. The use of quantum calculations in machine learning enables the generation of a diverse dataset and ensures that learning is less affected by noise inherent to experimental databases. In this work, we propose a transfer learning approach for the prediction of solvation free energies that combines fundamentals from quantum calculations with the higher accuracy of experimental measurements. The employed model architecture is based on the directed-message passing neural network for the molecular embedding of solvent and solute molecules. A significant advantage of models pre-trained on quantum calculations is demonstrated for small experimental datasets and for out-of-sample predictions. The improved out-of-sample performance is shown for new solvents, for new solute elements, and for the extension to higher molar mass solutes. The overall performance of the pre-trained models is limited by the noise in the experimental test data, known as the aleatoric uncertainty. On a random test split, a mean absolute error of 0.21 kcal/mol is achieved. This is a significant improvement compared to the mean absolute error of the quantum calculations (0.40 kcal/mol). The error can be further reduced to 0.09 kcal/mol if the model performance is assessed on a more accurate subset of the experimental data.
1. Introduction
Chemical and materials datasets often suffer from scarcity, bias, limited chemical-space coverage, and experimental uncertainty. The paper applies transfer learning from quantum calculations to improve solvation free-energy prediction, especially for small and out-of-sample experimental datasets.
- Experimental chemical and materials databases can be scarce, biased toward certain components, limited in chemical-space coverage, and experimentally noisy.
- Quantum calculations can generate additional, diverse data that compensates for experimental-data bias and expands chemical-space coverage.
- The paper introduces inductive transfer learning from quantum-chemistry models to experimental-data models for solvation free-energy prediction.
- The study provides CombiSolv-QM with 1 million quantum-calculated solvent/solute combinations and CombiSolv-Exp with 10145 experimental combinations.
- Earlier models showed random-split RMSE values of 0.57 kcal/mol, increasing to 1.45 kcal/mol for solvent clustering and 1.61 kcal/mol for solute clustering.
- Transferability is evaluated for new solvents, solutes containing specific elements, and solutes outside the training molar-mass range.
2. Methods - databases
The study constructs complementary quantum and experimental solvation databases. The quantum database is larger and broader, while the curated experimental database is smaller and subject to measurement uncertainty.
- CombiSolv-QM: The quantum database covers solute molar masses from 2.02 g/mol to 1776.89 g/mol and can be extended by combining computed COSMO surfaces.
- CombiSolv-QM: CombiSolv-QM contains 1 million COSMO-RS solvation free energies from 284 solvents and 11029 solutes.
- CombiSolv-Exp: CombiSolv-Exp contains 10145 solvent/solute combinations for 291 solvents and 1368 solutes after curation.
- CombiSolv-Exp: Experimental entries with standard deviations above 0.20 kcal/mol were removed, although reported uncertainty can reach 1 kcal/mol for some components.
- Database comparison: The QM database contains more high-molar-mass solutes, reaching 1776.89 g/mol versus 551.49 g/mol experimentally.
- Database comparison: For 3164 overlapping combinations, COSMO-RS predicts experimental solvation free energies with RMSE 0.67 kcal/mol and MAE 0.40 kcal/mol.
3. Methods - machine learning
The model represents solvent and solute molecules separately with directed message passing, combines their embeddings, and predicts solvation properties through a feed-forward network. Transfer learning initializes experimental models from quantum-trained parameters.
- Molecular representation: A D-MPNN separately converts solvent and solute graphs into latent molecular representations.
- Molecular representation: SMILES or InChI identifiers are converted into molecular graphs with atom and bond feature vectors using RDKit.
- Property prediction: The solvent and solute embeddings, together with additional molecular features, are concatenated and passed to a feed-forward property-prediction network.
- Transfer learning: Quantum-trained model parameters initialize models subsequently refined on CombiSolv-Exp data, while experimental-only models provide the comparison.
- Transfer learning: During transfer learning, D-MPNN parameters are frozen and FFN parameters are optimized for 20 epochs.
4. Performance of the quantum machine learning model
The quantum model improves as the training dataset grows and learns solvent embeddings organized by chemically relevant features. Its predictions on experiments combine model error with the intrinsic error of the quantum calculations.
- Dataset size: 1.43/0.95 kcal/mol RMSE/MAE is obtained with 10^3 quantum-data points, improving to 0.10/0.05 kcal/mol with 10^6 points.
- Interpretation: The learned solvent embedding is a black-box representation rather than a direct map of exact physical properties, so interpretation requires caution.
- Solvent embeddings: The first two solvent-embedding principal components explain 41.9% of the variance.
- Solvent embeddings: PC1 separates solvents by hydrogen-bonding capability and polarity, while PC2 separates molecules within a class by hydrocarbon-backbone length.
- Experimental predictions: The QM ensemble predicts experimental measurements with RMSE 0.81 kcal/mol and MAE 0.47 kcal/mol.
- Experimental predictions: The expected prediction error is close to the sum of quantum-calculation error and model error: 0.40 kcal/mol and 0.05 kcal/mol MAE, respectively.
5. Transfer learning from quantum chemical calculations to experimental data
The study transfers quantum-chemistry model parameters to experimental solvation data, then fine-tunes them on complete and reduced datasets. Pre-training improves learning with scarce data, while experimental noise limits achievable accuracy.
- The transfer-learning workflow applies quantum-model parameters to complete and subsetted CombiSolv-Exp datasets to assess performance at small dataset sizes.Models are fine-tuned on experimental data after pre-training on quantum calculations.
- At 1%, 5%, and 10% of the initial training data, RMSE decreases from 2.64, 1.45, and 1.14 to 0.76, 0.61, and 0.57 kcal/mol with pre-training.These correspond to 91, 457, and 913 data points, respectively.
- On random splits, both model types approach a MAE around 0.21 kcal/mol, because aleatoric uncertainty in experimental data limits prediction accuracy.The same noise also limits interpretation of model performance on the experimental test set.
- On a more accurate experimental subset, RMSE/MAE decreases from 0.44/0.21 to 0.14/0.09 kcal/mol for pre-trained models.The corresponding purely experimental values are 0.18/0.10 kcal/mol.
6. Transfer learning to improve out-of-sample predictions
Transfer learning is evaluated on solvents, solute elements, and higher-molar-mass solutes excluded from training. Pre-trained models generally improve out-of-sample predictions, especially where experimental coverage is sparse.
- 6.1. Solvent splits: For ethyl acetate, dichloromethane, and acetonitrile, pre-trained models improve performance over experimental models using structurally similar solvents.The improvement is associated with counterparts in the experimental database, though fewer than for common solvent classes.
- 6.1. Solvent splits: Pre-trained models outperform purely experimental models for left-out solvents with special structures, including THF, DMSO, and water.For all left-out solvents, pre-trained models outperform untuned QM models except water, where performance is similar.
- 6.1. Solvent splits: Transfer learning helps out-of-sample solvents because the D-MPNN learns chemical-structure similarities, but performance diminishes for poorly represented solvent structures.The paper reports that transfer learning can significantly improve performance in this regime.
- 6.2. Solute splits: For excluded solute elements, pre-trained models outperform both experimental and QM models after learning element representations from quantum data and fine-tuning experimentally.The gain is especially pronounced for elements with hydrogen-bonding accepting or donating sites, including O, N, and F.
- 6.2. Solute splits: For more similar halogens such as Cl, Br, and I, transfer-learning gains are smaller because experimental models can learn features such as electronegativity.
- 6.3. Molar mass splits: Across solute molar-mass cutoffs, pre-trained models outperform models trained solely on experimental data and exceed QM models at higher cutoffs.They perform similarly to QM models at 100 and 150 g/mol but outperform them at higher cutoffs.
7. Conclusions
The paper presents transfer learning from quantum chemistry to experimental solvation free-energy prediction, using a multi-molecule directed-message passing neural network. Pre-training improves performance for small experimental datasets and several out-of-sample settings, while larger-data accuracy is limited by experimental noise.
- Approach: Transfer learning combines quantum-chemistry pre-training with experimental-data refinement for solvation free-energy prediction.The architecture extends a directed message passing neural network to represent multiple molecules.
- Approach: The model is first trained on quantum data, then refined on experimental data while solvent and solute D-MPNN parameters remain frozen.
- Data scale: A clear transfer-learning advantage appears for experimental datasets containing up to 2000 data points.Fractions of the experimental database were used for training while predicting a fixed 10% test set.
- Out-of-sample performance: Pre-trained models outperform purely experimental models for left-out solvents, solute elements, and solute molar-mass splits.These results demonstrate improved out-of-sample performance across multiple chemically distinct splits.