Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

Qiwei Han; Chi Zhou

Source-linked AI summary

Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

Qiwei Han, Chi Zhou

arXiv:2610.12766v1Oct 8, 2026cs.LG

TL;DR

Random splits of the Perovskite Database can put identical devices in training and test sets, making it difficult to distinguish memorization from generalisation. The authors map institutional provenance and benchmark models using grouped and temporal splits; they report new state-of-the-art performance across splits and a shift toward neural models on unseen institutions and future discoveries.

Problem
Random splits can place identical devices in training and test sets, while prior evaluations used differing records and often lacked comparable baselines, limiting assessment of generalisation.
Source: Introduction · p. 2
Method
The authors map institutional provenance and evaluate thirteen models on grouped splits that isolate duplicate devices, publications, and institutions, alongside a temporal split.
Source: Conclusion · p. 13
Results
The benchmark establishes new state-of-the-art performance across splits, while neural architectures outperform tree-based models on unseen institutions and future discoveries.
Source: Introduction · p. 3, Conclusion · p. 13
Takeaways & Limitations
The authors recommend level 3 or temporal evaluation for generalisation claims, lookup-table controls for memorization, and joint consideration of absorber chemistry, device architecture, and fabrication.
Source: Conclusion · p. 13
In the temporal split, publication date is itself an input, so duplicates cannot cross a date boundary.
Source: Supporting Information · p. 16

Abstract

from arXiv · show

Machine-learning models for perovskite solar cells are typically evaluated using random data splits, which we show systematically leak information through massive internal duplication. Because nearly two-thirds of curated devices share their exact recipe with another, random splits primarily reward the mere recall of historical data rather than testing true physical generalisation. To solve this, we mapped the institutional provenance of the Perovskite Database and established a rigorous hierarchy of grouped splits that progressively isolate duplicate devices, publications, and institutions. Benchmarking thirteen models establishes new state-of-the-art performance across all splits and reveals a critical regime shift: while tree ensembles dominate random splits by recognising familiar recipes, their ranking collapses on genuinely unseen institutions and future discoveries, where composition-aware neural architectures excel. Furthermore, feature importance demonstrates that the device architecture and fabrication methodology dictate final efficiency just as strongly as the absorber chemistry. This standardised protocol empowers the community to separate memorisation from true physical discovery.

Introduction

The Perovskite Database contains more than 42,000 devices, but extensive recipe duplication means random splits can reward recall rather than learning how device properties relate to efficiency. The paper maps institutional provenance and introduces grouped and temporal evaluations, then benchmarks models across these splits.

The archive grew alongside rapid perovskite-cell progress, with efficiency rising from 3.8% to above 26% within a decade and records spanning thousands of publications.Source: Introduction · p. 1
Many records share identical inputs because studies report multiple similarly made cells and later studies replicate earlier designs, allowing duplicates into random-split training and test sets.Source: Introduction · pp. 1–2
Prior model evaluations used different record selections and mainly random splits, limiting direct accuracy comparisons and leaving duplicate recall and simple baselines unaccounted for.Source: Introduction · p. 2
The authors resolve institutional provenance for 94.7% of 4,683 publications, covering 796 institutions, and use it to enable strict unseen-institution evaluation.Source: Introduction · p. 2
A lookup table reaches 2.934% MAE against the reported state-of-the-art value of 2.936%, showing how random-split accuracy rewards duplicate recall.Source: Introduction · p. 2
The thirteen-model benchmark sets new state-of-the-art MAEs across random (1.973%), duplicate-separated (2.396%), publication (2.832%), institution (2.912%), and temporal (3.192%) splits.Source: Introduction · p. 3
Random-split model rank carries no information about new-institution rank; feature importance identifies absorber composition, device architecture, and fabrication process as predictive inputs.Source: Introduction · p. 3

Data Curation and Split Protocols

The curated archive retains 23,620 devices with complete core fields, expanded into descriptive features. The protocol groups records by recurring relationships and adds a date-based split to test forward in time.

Data Curation and Split Protocols

Curation retained 23,620 devices with complete core fields and expanded the archive records into 24 descriptive fields covering composition, device layers, fabrication, measurement, and performance.Source: Data Curation and Split Protocols · p. 3
The grouped hierarchy keeps identical device descriptions together at level 1, publications at level 2, and institutions at level 3; publication and institution serve only as split keys.Source: Data Curation and Split Protocols · pp. 3–4

Temporal split.

The temporal split trains on publications through 2016, validates on 2017, and tests on 2018 onward, assigning publications by date.Source: Temporal split · p. 4

Results and discussion

Grouped evaluation reveals that model performance and rankings change as duplicate, publication, and institutional overlap are removed. Tree ensembles lead on looser splits, while neural models outperform them on future publications; feature importance also implicates fabrication and device stack alongside composition.

Progressive elimination of data leakage

62.3% of curated devices share a description with another, and a lookup table reaches 2.643% MAE on a random split without learning device-efficiency relationships.Under the replicated state-of-the-art setup, the lookup table's MAE is 2.934%, against the reported 2.936% [16].Source: Progressive elimination of data leakage · pp. 5–6
Random splits leave 56.9% of test devices with a training duplicate; level 1 removes duplicates, level 2 separates publications, and level 3 separates institutions.At level 2, 93.0% of test devices still share an institution with training; level 3 removes duplicate, publication, and institutional overlap.Source: Progressive elimination of data leakage · pp. 5–6
Allowing same-publication devices into training lowers MAE by 0.215% ± 0.008%, while allowing same-institution devices lowers it by 0.106% ± 0.008%.The authors attribute these advantages to models implicitly learning source-specific systematic biases.Source: Progressive elimination of data leakage · p. 7

The thirteen-model benchmark

Random-split model rankings have weak or negative agreement with grouped-split rankings: Spearman correlations are +0.170 at level 2 and −0.159 on the temporal split.Across additional partitions, random-to-level 2 and random-to-level 3 rank correlations are −0.21 and −0.25.Source: The thirteen-model benchmark · p. 9
At level 3, SciBERT+attention has the lowest error (2.912%); on the temporal split, every neural model beats the gradient-boosted baseline by 0.19% to 0.27% and every tree-based model.Both temporal splits place neural models ahead of tree ensembles.Source: The thirteen-model benchmark · pp. 8, 10
Tree ensembles dominate looser splits by recognizing familiar layer-name categories, but SciBERT+attention leads at level 3 by reading layer-name semantics.At level 3, it beats CatBoost by 0.022% ± 0.057% and the gradient-boosted baseline by 0.105% ± 0.071%.Source: The thirteen-model benchmark · p. 10
On later publications, neural models predict efficiencies up to 21.8%, while tree predictions do not exceed 19.4%; the temporal test devices average 13.59% PCE versus 10.54% in training.The authors attribute the divergence to tree predictions being constrained by training-era outcomes; removing each model's mean bias brings the model families level.Source: The thirteen-model benchmark · pp. 10–11

Feature importance: from labels to physical properties

At level 3, removing fabrication process, device stack, and absorber composition raises test MAE by 0.091%, 0.082%, and 0.074%, respectively.On random splits, removing the device stack raises error by 0.342%; layer identity's contribution falls from 0.236% to 0.026% at level 3 as leakage is removed.Source: Feature importance: from labels to… · pp. 11–12

Conclusion

Random splits on the Perovskite Database can leak information through internal duplication, making memorization difficult to distinguish from physical understanding. The authors establish provenance-based grouped and temporal evaluation, finding that neural models outperform tree-based models on unseen institutions and future discoveries.

Conclusion

Random splits systematically leak information through internal duplication, inflating apparent performance through recall of historical recipes.The conclusion frames this as mistaking memorization for physical understanding.Source: Conclusion · p. 13
A provenance-based framework progressively closes leakage pathways by separating publications and institutions and testing forward in time.The authors release the curated data, mapped institutional provenance, grouped splits, and benchmarked suite of 13 models.Source: Conclusion · p. 13
Neural architectures outperform tree-based models in extrapolation to unseen institutions and future discoveries.The conclusion recommends level 3 or temporal evaluation, tree ensembles as baselines, and lookup-table controls for memorization.Source: Conclusion · p. 13
The authors recommend co-engineering device architecture and fabrication methodology alongside absorber chemistry.This recommendation follows their conclusion that all three affect final efficiency as strongly as the others.Source: Conclusion · p. 13

Supporting Information

The supporting information documents the curated archive, device-field construction, split definitions, and additional benchmark and feature analyses.

Supporting Information

The curated archive's field-missingness summary covers 19 core device fields, including performance outputs, composition, device layers, architecture, area, annealing, and deposition.The figure description gives missing-value percentages for the original 42,443 archived records.Source: Supporting Information · p. 14
The device-field documentation distinguishes archived from derived fields, including absorber composition, layer sequences, fabrication descriptors, cell geometry, and measured outputs.The listed measured outputs include power conversion efficiency, open-circuit voltage, short-circuit current density, and fill factor.Source: Supporting Information · pp. 14–15
The split table reports test-device counts, test-unit counts, and the shares of test devices overlapping training by duplicates, publication, or institution.For level 3, the supplied values show 0.0% overlap for each of these three categories.Source: Supporting Information · pp. 16–17
The supplementary analyses describe model-error comparisons, temporal test sets, feature-group removal, and permutation-based feature reliance across evaluation splits.The temporal comparisons use training publications through 2016 or 2017 and test publications from 2018 or 2019 onward.Source: Supporting Information · pp. 17, 19, 20

Methods

The methods curate records using completeness and physical-validity filters, then construct model inputs from structured device fields. Tree-based and tabular neural models use encoded columns, while text models process recorded layer names with a frozen scientific language model.

Data curation.

Of 42,443 database records, 23,620 survive three curation stages requiring complete core fields and excluding implausible, unparseable, or physically unusable records.The retained records are expanded into 24 descriptive fields.Source: Data curation · p. 21

Model inputs.

Tree-based models use 40 columns, with text categories target- and count-encoded; out-of-fold training encodings prevent a device's own label from entering its encoding.The four tabular neural models receive the same columns plus absorber weights, with missing values set to training medians and a training-fitted quantile transformation.Source: Model inputs · p. 21
Two text models process device-layer text with frozen SciBERT embeddings, using either mean pooling or learned attention over token states.The encoder is applied once per text and cached; because it is frozen, nothing is fitted to the corpus.Source: Model inputs · p. 22

Provenance mapping.

Publication provenance is mapped to corresponding-author institutions, while duplication analysis identifies groups sharing the same 24 descriptive fields and checks their full device records.

Provenance mapping.

OpenAlex resolves 99.9% of 4,683 publications and identifies corresponding-author institutions for 94.7%, covering 796 institutions and 95.8% of devices.Unresolved publications and those without a corresponding author remain unassigned.Source: Provenance mapping · p. 22

Duplication analysis.

Devices are duplicates when all 24 descriptive fields agree; groups are then checked across 249 archive columns describing device fabrication.The 249 columns exclude measured outcomes and bibliographic fields.Source: Duplication analysis · p. 22
Within-group efficiency spread is measured as each duplicated device’s PCE minus its group’s mean PCE.This spread is shown in Fig. S1.Source: Duplication analysis · p. 22

Split definitions.

The evaluation uses five shared splits that progressively group devices by duplicates, publication, institution, or time. Two recall controls predict from matching or nearby training devices rather than learning input–efficiency relationships.

Split definitions.

Five splits assign whole units to training, validation, and test: individual devices, duplicate groups, publications, institutions, or publication dates.The random split is stratified by PCE quintile at 80/10/10; level 1 groups duplicates, and levels 2 and 3 hold out publications and institutions, respectively.Source: Split definitions · pp. 22–23

Temporal split.

The temporal split trains on publications through 2016, validates on 2017, and tests from 2018, with a second framing testing from 2019.A publication has one date and therefore does not cross the temporal boundary; the second framing trains through 2017 and validates on 2018.Source: Split definitions · p. 23

Split definitions.

Partitions use seed 42, with ten additional seeded partitions, and target normalisation uses training devices only.The published seed-42 partitions are exactly reproduced by rebuilding them with the stated rules.Source: Split definitions · p. 23

Recall controls.

Two recall controls predict from training devices: a lookup table uses the median PCE for identical inputs, while nearest neighbours average the closest k devices.If no identical input exists, the lookup table returns the training median; k is selected using validation MAE.Source: Recall controls · p. 23

Models.

The study compares a gradient-boosted baseline with twelve models and uses configurations selected on level 1 validation data. Configurations are reused across splits without test-set tuning or selection.

Models.

Twelve comparison models span boosted trees, other tree ensembles, MLP variants, transformer models, TabM, and SciBERT-based models.CatBoost is evaluated with out-of-fold target encoding and native categorical handling; a lighter boosted reference model is used for repeated refitting analyses.Source: Models · p. 23
The baseline is HistGradientBoostingRegressor with absolute-error loss, 1,500 maximum iterations, 63 leaves, and early stopping.Its configuration was selected against level-1 validation error and averaged over three seeds.Source: Models · p. 23, Benchmark settings · p. 23

Benchmark settings.

Each comparison model is tuned on level 1 alone, then evaluated with that configuration across splits; test devices are not used for tuning, stopping, or model selection.Three seeds are averaged, and the models share the same test devices.Source: Benchmark settings · pp. 23–24
The Solar-GECO setup is replicated from its description using five device-layer names and ten random 80/10/10 splits.The processed set contains 28,922 devices from 5,565 publications; code and processed data from the original study were unavailable.Source: Benchmark settings · p. 24

Leakage analysis.

The leakage analysis varies related-device exposure while holding test devices and training-set size fixed, and separately controls for smaller training sets. Temporal evaluation tests a second publication-date framing and defines test-device novelty relative to training data.

Leakage analysis.

Related-device comparisons hold test devices and training-set size fixed, remove duplicates, and repeat each comparison twenty times.Publication comparisons use halves of each level-2 test publication; institution comparisons divide publications within eligible level-3 test institutions into halves and exclude test publications from training.Source: Leakage analysis · p. 24
Training-size controls shrink level-1 training data to level-2 size and level-2 training data to the 15,535-device level-3 size.The first control uses three splits; the second uses three level-2 partitions and five random subsets each.Source: Leakage analysis · p. 24

Temporal split.

A second temporal framing trains through 2017, validates on 2018, and tests from 2019 to examine prediction across a two-year gap.A test device is new in a field when that field value does not occur in the training devices, or in training and validation devices; institution shares count known-institution test devices.Source: Temporal split · p. 25

Feature importance.

Feature importance is assessed by removing or shuffling provenance-defined input groups, while uncertainty is estimated with methods that respect the split’s grouping unit.

Feature importance.

Feature-group removal retrains the reference model without one group and measures the test-MAE increase as a share of the full model’s gain over the training-median baseline.Inputs are grouped by column provenance, separating layer thickness from layer identity.Source: Feature importance · p. 25
Shuffling fixes the trained baseline and randomly reassigns composition and bandgap, device stack, or fabrication inputs among test devices; its magnitudes are compared across splits, not with removal.Publication year and cell geometry are not shuffled.Source: Feature importance · p. 25

Uncertainty estimation.

Test-result uncertainty uses a paired bootstrap over each split’s grouping unit, with 20,000 resamples; uncertainty across repeated splits or repetitions uses the standard error of the mean.Training-seed variation is reported separately and not pooled with test uncertainty.Source: Uncertainty estimation · pp. 25–26

Data and code availability.

The reproducibility release provides split assignments, provenance and model outputs, plus code and scripts for reconstructing the processed data and reproducing analyses.

Data and code availability.

The release includes row-level assignments for every split and its ten further partitions, publication and resolved-institution provenance for every device, and each model’s optimization trace, selected configuration, and test predictions.It also includes scored result tables and analysis code from dataset construction through interval reproduction.Source: Data and code availability · p. 26
The Perovskite Database source file is not redistributed; replay scripts reconstruct the processed data and splits bit-identically from the public source.Source: Data and code availability · p. 26
The released split assignments and device provenance are sufficient to place a new model on the same protocol without retraining the authors’ models.Source: Data and code availability · p. 26