PaperlayerView exact revision on arXiv

Source-linked AI summary

The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning

Kareem M. Gameel, Ihor Neporozhnii, Sjoerd Hoogland, Oleksandr Voznyy

arXiv:2609.28782v1cs.LGphysics.chem-phphysics.comp-ph

TL;DR

Delta-learning often assumes that more accurate baselines leave smaller residuals that are easier to learn, but baseline accuracy and residual scale alone do not determine residual learnability. This paper evaluates target transformations through residual scale and architecture-informed smoothness, finding that local descriptor baselines can leave rougher residuals while xTB reduces scale and roughness and improves prediction.

  • Problem

    Delta-learning often assumes that more accurate baselines leave smaller, easier residuals, although reported outcomes are contradictory and residual learnability remains difficult to assess before training.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    The paper evaluates delta-learning target transformations by residual scale and architecture-informed smoothness, introducing IQR-normalized graph Dirichlet roughness as a candidate pre-training diagnostic.

    Source: §1 · p. 2

  • Results

    Local-descriptor MLP baselines leave smaller but rougher residuals; xTB reduces scale and roughness while improving in-domain accuracy and out-of-domain transfer, with DFTB as a supporting control.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Baseline transformations should be assessed jointly by residual scale and scale-normalized smoothness, with D_IQR considered as a candidate pre-training diagnostic.

    Source: §1 · p. 2, §4 · p. 12

    The roughness–n-MAE association does not establish causality or a universal error law in the small, related target panel.

    Source: Supplementary Note 7: Normalization… · p. 31

Abstract

from arXiv · show

In scientific machine learning, $\Delta$-learning trains models on residual errors relative to physical baselines, assuming that more accurate baselines with smaller residual scales inherently improve downstream performance. Here, we demonstrate that residual scale alone is an insufficient heuristic for learnability. Evaluating molecular graph neural networks on total energy targets, we show that complex local descriptor baselines can yield small residual targets that are disproportionately rough within architecture-informed proxy spaces and harder to learn relative to their scale. Conversely, semi-empirical baseline reduces both scale and normalized roughness, improving in-domain and out-of-domain prediction. We introduce scale-normalized graph Dirichlet roughness ($D_{\text{IQR}}$) as a pre-training diagnostic for residual learnability and establish baseline complementarity as a core target-design principle, elevating target space formulation alongside model architecture as a key axis for scientific machine learning.

1 Introduction

Scientific machine learning often treats targets as fixed while refining representations, but heterogeneous molecular energies can vary unevenly across a fixed GNN feature space. This paper tests target design through residual scale and architecture-informed smoothness, finding that residual scale alone does not predict learnability.

1 Introduction

  • Heterogeneous targets such as molecular total energies combine contributions spanning interaction orders and length scales that a fixed GNN may represent with different accuracy.Poorly aligned components may increase target roughness in the model’s feature space and limit generalizability.Source: §1 · p. 2
  • The proposed target-design approach considers both target scale and smoothness in an architecture-informed representation space, since delta transformations alter scale and functional form.At fixed normalized error, shrinking target scale reduces absolute error, while changes in functional form can alter scale-normalized learnability.Source: §1 · p. 2
  • Delta-learning is often expected to leave smaller, easier residuals with more accurate or better-aligned baselines, but reported studies also find negative results or better performance from less accurate baselines.These contrasting outcomes motivate assessing residual learnability relative to model inductive bias, not baseline accuracy or residual scale alone.Source: §1 · p. 2
  • IQR-normalized graph Dirichlet roughness, D_IQR, is introduced as a candidate pre-training diagnostic of target variation across a fixed architecture-informed feature space.Low roughness means nearby molecules in feature space have similar target values relative to scale.Source: §1 · p. 2
  • The framework is evaluated on molecular total energies with GemNet-T and SchNet, using tmQM for in-domain performance and QM9 for out-of-distribution transfer.The architectures contrast directional and angular message passing with pair-distance representation, and the evaluation spans baselines with varied physical priors and feature-space alignment [14].Source: §1 · p. 2
  • Across both architectures, MLP descriptor baselines leave smaller but rougher residuals, whereas xTB reduces scale and roughness while improving in-domain accuracy and out-of-domain transfer.DFTB also improves prediction despite slightly increased normalized roughness.Source: §1 · p. 2

2 Results

Baseline accuracy and residual scale alone do not determine GNN learnability: local descriptor baselines can leave smaller but rougher residuals, while xTB improves in-domain and out-of-domain prediction. Architecture-informed, scale-normalized roughness is evaluated as a pre-training diagnostic of relative learnability.

2.1 Target shrinkage does not guarantee learnability

  • Linear BoE cuts IQR 10.9-fold while raising n-MAE 7.7-fold; MLP-BoE shrinks IQR 1.38-fold but raises n-MAE 3.27-fold and absolute MAE 2.37-fold.Thus, further target shrinkage does not necessarily improve downstream learning.Source: Figure 1 · p. 3, §2.1 · pp. 3–4

2.2 Pre-training roughness quantifies model-relative learnability

  • DIQR rose from 0.548 for Eref to 1.127 for linear-BoE and 1.658 for MLP-BoE, matching the post-training n-MAE ordering.The IQR-normalized graph roughness measures target variation over architecture-informed proxy-graph edges and is proposed as a pre-training learnability diagnostic.Source: §2.2 · p. 5

2.3 Residual roughness guides target design beyond baseline accuracy

  • xTB left a larger residual than BoB, PRE, and PRE+A, yet reduced DIQR from 1.127 to 0.964 and mean GemNet test MAE from 0.00945 to 0.00660 Ha.Among primary baselines, xTB was the only one that lowered both normalized roughness and GemNet test MAE.Source: Figure 2 · p. 7, §2.3 · p. 7
  • BoB, PRE, and PRE+A reduced residual IQR by factors of 3.3, 4.2, and 4.7, but raised DIQR by factors of 3.3, 9.5, and 7.6, respectively.None improved GemNet test MAE over direct training on Eref,BoE.Source: §2.3 · p. 6, Figure 2 · p. 7

2.4 Model-relative target roughness predicts scale-normalized learnability

  • Across 13 targets, GemNet n-MAE increased approximately linearly with log DIQR (R2 = 0.905), though shared IQR normalization contributes substantially to the association.The results support DIQR as a candidate pre-training diagnostic, but do not establish causality or guarantee prediction for new target families.Source: Figure 3 · p. 8, §2.4 · pp. 8–9

2.5 Target design controls out-of-domain transfer

  • On QM9, xTB-residual GemNet reduced mean-shifted MAE from 0.01641 to 0.00598 Ha and raw MAE from 0.2478 to 0.0248 Ha, while Kendall τ rose from 0.726 to 0.901.Among primary GemNet routes, the composition-referenced xTB residual had the best mean transfer performance across all three reported metrics.Source: Figure 4 · p. 10, §2.5 · p. 10

3 Discussion

Baseline accuracy and residual scale alone do not determine downstream learnability: local descriptor residuals can be small yet rough, while semiempirical baselines can improve prediction. The discussion frames scale and architecture-relative smoothness as complementary target-design considerations, while treating the proposed mechanism and transfer interpretation cautiously.

Baseline complementarity

  • Local descriptor residuals are smaller but disproportionately rough, and their MLP predictions are poorer than direct DFT-BoE learning, consistent with—but not proving—a roughness-related generalization limitation.The interpretation does not rule out optimization or network-capacity effects.Source: §3 · p. 11
  • The xTB baseline reduces both target scale and normalized roughness while improving test performance; DFTB lowers matched-subset MAE despite slightly higher normalized roughness.Thus, roughness reduction is not necessary for lower absolute error.Source: §3 · p. 11
  • The observed baseline trends are consistent with complementarity: local baselines may remove variations message-passing layers already represent, whereas semiempirical baselines may absorb contributions local architectures represent poorly.The authors present this as a physical interpretation, not an identified causal mechanism.Source: §3 · p. 11

Pre-training diagnostic

  • DIQR is a candidate pre-training screening diagnostic because it strongly associates with normalized test error, but the test-set analysis is retrospective and prospective screening needs an independent evaluation set.Shared normalization contributes substantially to the association, with permutation controls supporting an additional contribution from molecular-neighbourhood alignment.Source: §3 · p. 11

Out-of-domain transfer

  • On QM9, the xTB residual supports transfer across distinct chemical environments, but these results do not establish that roughness alone predicts transfer or that learned representations improve.The xTB residual showed superior trend preservation and energetic rank correlation relative to local descriptor baselines.Source: §3 · pp. 11–12

4 Conclusion

The conclusion argues that target transformations should be assessed by both residual scale and scale-normalized representation smoothness, rather than scale reduction or baseline accuracy alone. It positions DIQR as a candidate pre-training diagnostic and target design as a complement to architecture design.

4 Conclusion

  • Residual scale and scale-normalized representation smoothness provide complementary diagnostics, and DIQR is supported as a candidate pre-training diagnostic within a target-design framework.The paper identifies extensions to diverse physical properties, multi-fidelity workflows, and machine-learning interatomic potentials.Source: §4 · p. 12

5 Methods

The methods define aligned tmQM targets and baseline residuals, compare descriptor and semiempirical baselines with fixed downstream training protocols, and evaluate roughness and transfer using fixed splits and test cohorts. Roughness associations are retrospective diagnostics, while QM9 evaluation uses frozen tmQM-fitted baselines.

5.1 Datasets and targets

  • The study uses aligned neutral tmQM complexes with fixed splits, atom- and composition-referenced targets, and common test molecules; QM9 is reserved for OOD evaluation at the same electronic-structure level.DFTB failures were not imputed, and its successful-calculation subset has separate counts.Source: §5.1 · p. 12

5.2 Baseline models

  • Baselines include BoE, a nonlinear composition-only control, study-specific BoB, PRE, and PRE+A descriptor models, plus fixed semiempirical xTB and DFTB baselines.Descriptor baselines use ridge and MLP fits; baseline fitting and preprocessing use training data, with validation selecting MLP checkpoints.Source: §5.2 · pp. 12–13

5.3 GNN training

  • GemNet-T is the primary model and SchNet the architecture control; both use fixed target-wise protocols, with 80-epoch training and seed-level test and QM9 evaluation.Primary targets use three independent seeds; DFTB targets use one seed, so they have no across-seed uncertainty estimate.Source: §5.3 · p. 13

5.4 Roughness and transfer diagnostics

  • Architecture-specific fixed proxy graphs support DIQR measurement, and semi-log regressions relate ln DIQR to normalized test error across 13 primary target means.These retrospective test-set diagnostics do not use test labels for baseline fitting, training schedules, or checkpoint selection; leave-one-target-out fits are influence checks, not cross-validated prediction scores.Source: §5.4 · pp. 13–14

5.5 OOD transfer evaluation

  • For OOD transfer, tmQM-trained models are evaluated on QM9 without retraining, using frozen training-only baselines and 128,988 common molecules for primary parent-energy comparisons.Mean-shifted MAE uses an offset estimated from evaluated QM9 reference labels and is an oracle diagnostic, not an independently calibrated prediction score.Source: §5.5 · p. 14

Scope and Reporting Conventions

The supplement compares SchNet and GemNet across a shared target panel, reporting seed-level performance and documenting cohort and control conventions. Target-specific training cohorts vary, while the architectures are matched within each target.

Scope and Reporting Conventions

  • The supplement reports 78 main runs from three independent seeds, plus four single-seed DFTB runs across two targets and two architectures.Each run is evaluated on the tmQM test split and on QM9 using its own best-validation checkpoint.Source: Scope and Reporting Conventions · p. 18
  • DFTB is a supporting, single-run control excluded from pooled analyses, and its success-filtered cohort is separate.Uncertainty denotes sample standard deviation across independently scored seeds; predictions are not ensembled.Source: Scope and Reporting Conventions · p. 18
  • The 13-target panel distinguishes linear and nonlinear BoE references and descriptor residuals, alongside referenced xTB energies and their DFT residuals.An unsubscripted BoE reference is linear; BoE-MLP is fitted directly from the DFT parent target.Source: Scope and Reporting Conventions · p. 18
  • Both architectures use identical target values and split memberships within each target, but training cohorts differ across some targets.The six descriptor residuals use 56,821 training and 7,103 validation molecules; other main targets use 56,830 and 7,104, and all 13 share 7,104 test molecules.Source: Scope and Reporting Conventions · p. 19

Supplementary Section 1: Target Preprocessing and Scale Controls

Linear BoE referencing reduces downstream errors, whereas a nonlinear BoE can narrow residuals while making them harder to predict. Scale normalization removes uniform amplitude changes, but fitted baselines may alter residual structure.

Supplementary Note 1: Direct Preprocessing Controls

  • Nonlinear BoE narrows residuals further, yet test MAE rises to 0.024672 ± 0.000112 Ha for SchNet and 0.022380 ± 0.000452 Ha for GemNet.Thus, smaller residual distributions do not guarantee easier regression.Source: Supplementary Note 1: Direct… · p. 20
  • 0.019917 ± 0.000178 to 0.015648 ± 0.000077 Ha for SchNet, and 0.013360 ± 0.000423 to 0.009448 ± 0.000469 Ha for GemNet, after linear BoE correction.Figure S1 compares target IQR with tmQM test MAE.Source: Figure S1 · p. 20, Supplementary Note 1: Direct… · p. 20

Supplementary Note 2: What Scale Normalization Removes

  • Standardized test-target distributions are centered by their test median and divided by their test IQR; IQR-normalized roughness is unchanged by nonzero uniform rescaling.A fitted baseline can instead change molecular ordering, tail structure, and local differences.Source: Supplementary Note 2: What Scale… · p. 21, Figure S2 · p. 22
  • The invariance is algebraic: it does not establish identical normalized errors for independently optimized networks, and shared scale factors can contribute to diagnostic correlations.The supplement examines that correlation issue separately.Source: Supplementary Note 2: What Scale… · p. 22

Supplementary Section 2: Baseline-Selection Diagnostics

Baseline accuracy alone does not determine downstream performance: xTB gives the lowest mean tmQM error in the panel, while descriptor MLP residuals can remain difficult to learn. Small residual scale does not ensure better prediction or transfer.

Supplementary Note 3: Baseline Accuracy and Downstream Error

  • BoB, PRE, and PRE+A MLP baselines have smaller baseline errors than their linear counterparts, yet none beats direct DFT-BoE learning for GemNet.For SchNet, linear BoB improves on direct learning, while linear PRE and PRE+A do not improve mean test MAE.Source: Supplementary Note 3: Baseline… · p. 23
  • 0.009569 ± 0.000101 Ha for SchNet and 0.006596 ± 0.000048 Ha for GemNet are the xTB residuals’ lowest mean tmQM test MAEs in the 13-target panel.These errors are respectively 1.635-fold and 1.432-fold lower than direct DFT-BoE learning; this is an empirical result, not proof of a particular mechanism.Source: Supplementary Note 3: Baseline… · p. 24

Supplementary Note 4: Descriptor Baseline Capacity

  • MLP-PRE and MLP-PRE+A concentrate residuals near zero, but downstream errors and roughness do not decrease in proportion to IQR.Their in-domain performance should not be extrapolated to QM9, where fixed descriptor baselines can behave differently.Source: Supplementary Note 4: Descriptor… · p. 24

Supplementary Section 3: QM9 Transfer Controls

The transfer evaluation reports native-target scores and reconstructs a common DFT-BoE parent for matched comparisons. On that cohort, xTB residual routes retain lower oracle-shifted errors than descriptor-MLP residuals, whose small in-domain widths do not ensure good transfer.

Supplementary Note 5: Complete Transfer Evaluation

  • The common-parent analysis evaluates main DFT-prediction routes on the same 128,988 QM9 molecules; DFTB uses a separate 128,978-molecule intersection.Direct xTB and DFTB energy predictors are not DFT-prediction routes in these parent-energy tables.Source: Supplementary Note 5: Complete… · p. 25
  • Oracle mean-shifted MAE uses QM9 reference labels, is not deployable as an uncalibrated prediction, and removes constant bias but not ordering errors, heteroscedastic scatter, or outliers.Native-target cohorts differ, so their scores are not exact all-target matched-cohort comparisons; raw transfer errors remain essential.Source: Supplementary Note 5: Complete… · p. 25

Parent-Energy Reconstruction and Trend Fidelity

  • MLP-PRE and MLP-PRE+A have oracle-shifted errors near 0.061–0.066 Ha and negative parent-energy R2 despite small in-domain residual widths.PRE+A can retain moderate Kendall ordering while having poor RMSE and R2; rank fidelity does not protect against large amplitude errors or outliers.Source: Parent-Energy Reconstruction and Trend… · p. 28
  • BoE-referenced xTB gives approximately 0.00665 Ha for SchNet and 0.00598 Ha for GemNet in oracle-shifted error, with Kendall τ near 0.895 and 0.901, respectively.The comparison concerns reconstructed parent-energy transfer on the common main cohort.Source: Parent-Energy Reconstruction and Trend… · pp. 27–28, Figure S5 · p. 28

Supplementary Section 4: Architecture Controls and Roughness

Architecture-proxy roughness tracks normalized test error across the main target panel, but the association depends partly on shared normalization. Permutation and similar-IQR controls support a geometry-sensitive component without establishing causality or a universal error law.

Supplementary Note 6: Matched Architecture Panels

  • The primary graphs use 7,104 shared test molecules, cosine k = 10 neighbours, and locally scaled Gaussian edge weights; roughness and error use the same test cohort and IQR.Test labels supply target differences and IQR after models are frozen; this is a retrospective diagnostic, not a prospectively validated model-selection rule.Source: Supplementary Note 6: Matched… · p. 29, Table S8 · p. 30
  • Kendall τ across the 13 target means is 0.872 for SchNet and 0.821 for GemNet, but 39 runs per architecture are not 39 independent target-design observations.The roughness analysis uses fixed geometric proxies: PRE for SchNet and PRE+A for GemNet, not learned network representations.Source: Supplementary Note 6: Matched… · pp. 29–30, Table S8 · p. 30

Supplementary Note 7: Normalization and Panel Sensitivity

  • With raw error, R2 falls to 0.060 for SchNet and 0.332 for GemNet; with neither axis IQR-normalized, R2 is 0.067 and 0.019.The normalized association should not be read as equally strong evidence that unnormalized roughness independently predicts absolute error.Source: Table S9 · p. 30, Supplementary Note 7: Normalization… · p. 31
  • Residualizing ln DIQR and ln n-MAE against ln IQRtest gives partial Pearson correlations of 0.640 and 0.690, but shared-denominator coupling remains and the related target panel limits interpretation.These correlations check association beyond a linear log-scale trend, not causality or a universal error law.Source: Supplementary Note 7: Normalization… · pp. 30–31
  • Across 5,000 joint label permutations, observed ln DIQR–n-MAE correlations exceed every shuffled value (one-sided plus-one p = 1/5001 ≈ 0.00020), conditional on this panel and scheme.High shuffled median R2 values—0.862 for GemNet and 0.855 for SchNet—show that target distributions and normalization contribute substantially; the result does not establish causality or a universal law.Source: Supplementary Note 7: Normalization… · pp. 31–32, Table S10 · p. 32
  • The 10% IQR-matching threshold selects three pairs, all concordant in roughness and normalized-error ordering for both models, but matching does not eliminate denominator differences.Linear PRE+A has an IQR 8.4% larger than MLP-BoB, yet lower roughness and normalized error.Source: Supplementary Note 7: Normalization… · p. 32, Table S11 · p. 32

Supplementary Section 5: DFTB Semiempirical Control

DFTB is evaluated as a supporting semiempirical control on molecules with successful calculations. On the matched valid subset, its residual is rougher than the DFT parent despite improving matched-subset test MAE, while the comparison remains qualified by differing training cohorts and one seed.

Supplementary Note 8: DFTB Coverage and Matched Evaluation

  • DFTB calculations failed for 362 of 71,038 neutral tmQM molecules, so valid splits contain 56,540 training, 7,068 validation, and 7,068 test molecules.Failed energies were not imputed.Source: Supplementary Note 8: DFTB Coverage… · p. 33
  • On common DFTB-valid test molecules, residual DIQR is 1.184 in PRE and 1.169 in PRE+A, versus 1.127 and 1.113 for the DFT parent; matched-subset test MAE nevertheless improves.SchNet test MAEs are 0.013980 Ha for DFTB energy and 0.012119 Ha for its residual; GemNet values are 0.008462 and 0.007936 Ha.Source: Supplementary Note 8: DFTB Coverage… · p. 33, Table S12 · p. 33
  • The matched evaluation uses identical DFTB-valid test or QM9 molecules, but training cohorts differ; no exact-mask direct-DFT GNN or three-seed estimate is included.The control supports a qualified comparison, not an unrestricted DFTB ranking.Source: Supplementary Note 8: DFTB Coverage… · p. 33

Supplementary Section 6: Independent-Seed Results

The independent-seed analyses distinguish GNN training variation from baseline-fitting variation. SchNet’s linear PRE and PRE+A runs show larger test variability than most other targets, while alternative MLP fitting seeds yield nonidentical residual labels not covered by the GNN error bars.

Supplementary Note 9: GNN Seed Variation

  • The reported SD measures training-seed variation for a fixed dataset and target, not uncertainty from new molecules, baseline fitting, or a different domain.Deterministic IQR and roughness values have no GNN-seed error bars.Source: Supplementary Note 9: GNN Seed Variation · p. 34
  • SchNet linear PRE and PRE+A seed-102 test MAEs are 0.017296 and 0.017444 Ha, versus approximately 0.0154–0.0156 Ha for the other seeds; these runs are retained.Mean-score differences comparable to this variation should not be described as decisive.Source: Table S13 · p. 35, Supplementary Note 9: GNN Seed Variation · p. 36

Baseline Fitting Seeds Are Separate

  • Alternative MLP baseline fits have test-MAE SD below 0.0006 Ha in each family, yet individual residual labels differ, with maxima of 1.093 Ha for PRE and 0.866 Ha for PRE+A.GNN error bars do not quantify this added variability; robustness would require downstream evaluation on alternative targets.Source: Baseline Fitting Seeds Are Separate · p. 37, Table S17 · p. 37

Supplementary Section 7: Supplementary Methods

The supplementary methods document the datasets, train-only baseline fitting, model configurations, and retrospective roughness analysis. They also delimit the statistical claims and provide an audit trail for predictions and released materials.

Datasets, References, and Splits

  • The benchmark uses 71,038 neutral tmQM molecules with DFT energies at ωB97M-V/def2-SVPD; frozen split identifiers and explicit molecule-ID joins preserve dataset alignment.QM9 uses the same DFT level, and filename “full” denotes concatenated split rows evaluated with one frozen model, not a model trained on all molecules.Source: Datasets, References, and Splits · p. 38

Train-Only Baseline Fitting

  • Train-only baseline fitting includes linear BoE models from 44 elemental counts and descriptor baselines; final coefficients and preprocessing are fitted on training data and frozen for evaluation.BoB is a 4,950-dimensional typed-bond-count descriptor; PRE has 34,650 pair-radial features, while PRE+A adds angular features for 45,738 total.Source: Train-Only Baseline Fitting · p. 38

Semiempirical Targets

  • GFN2-xTB supplies the xTB baseline, while DFTB+ uses SCC-DFTB/DFTB2 with PTBP parameters and successful-calculation filtering.The DFTB parameterization is used as a control, not claimed to be optimal for the molecular domain.Source: Semiempirical Targets · pp. 38–39

GNN Configurations and Selection

  • SchNet and GemNet-T use energy-only learning with a 6.0 Å cutoff and no force regression; validation controls scheduling and checkpoint selection, not test or QM9 labels.Each target is standardized using its own training mean and population SD, shared across architectures and seeds.Source: GNN Configurations and Selection · p. 39, Table S18 · p. 39

Roughness and Statistical Scope

  • Roughness uses fixed architecture-informed test graphs, and the primary association is descriptive across 13 related targets; normalization controls and permutations do not establish causality or guaranteed prospective ranking.Targets are not assumed independent, and DFTB is excluded from pooled calculations.Source: Roughness and Statistical Scope · pp. 39–40

Prediction Audit and Reproducibility

  • The audit verifies 82 runs, prediction coverage, recomputed MAEs, split checksums, and checkpoint paths; the corrected release includes ledgers, roughness data, and plotting code but omits raw structures and model checkpoints.Rebuilding the SI does not retrain models or change the main manuscript.Source: Prediction Audit and Reproducibility · p. 40
Loading 2609.28782v1…