Source-linked AI summary
Few-Shot Bioactivity Prediction with Meta-Learning under Assay Heterogeneity
Michal Kmicikiewicz, Tommy Rochussen, Vincent Fortuin, Ewa Szczurek
TL;DR
Individual bioactivity assays often have too few measurements for reliable standalone prediction, while meta-learning depends on sufficient relatedness among training tasks. This paper introduces MetaHeta, which conditions predictions on related-assay data, and reports that greater task heterogeneity can impair meta-learning while MetaHeta improves prediction and compound prioritization in tested settings.
Problem
Bioactivity assays often lack enough measurements to train reliable independent predictors, while sharing information across assays risks conflating distinct assay-specific relationships.
Source: §1 · p. 1Method
MetaHeta conditions predictions on auxiliary data from related assays, using exact attention for scarce task-specific context and linear attention for large auxiliary datasets.
Source: §1 · p. 2Results
Greater meta-training task heterogeneity can degrade meta-learning performance, while MetaHeta improves few-shot prediction and downstream compound prioritization on ChEMBL and BindingDB assays.
Source: §1 · p. 2, §5.1 · p. 7, §6 · p. 10Takeaways & Limitations
In the tested settings, adding heterogeneous meta-training tasks need not improve few-shot prediction, whereas conditioning on related assays is effective across shared-target, target-homology, and assay-description notions of relatedness.
Source: §5.1 · p. 7, §6 · p. 10The theoretical results motivating auxiliary-assay selection do not directly guarantee gains for finite-capacity models trained on finite meta-datasets, and description-based relatedness is heuristic.
Source: §E.1 · p. 24
Abstract
from arXiv · showhide
Accurate bioactivity prediction is a central challenge in early-stage drug discovery, as individual assays often contain too few measurements to train reliable models independently. Meta-learning offers a principled approach to this few-shot setting, but assay heterogeneity may limit its effectiveness. Here, we test this hypothesis and show that meta-learning performance degrades as meta-training tasks become more heterogeneous. To address this, we introduce MetaHeta, a meta-learning framework that accounts for assay heterogeneity by conditioning predictions on auxiliary data from related assays, with relatedness defined flexibly from available assay information. The architecture of MetaHeta combines linear attention over large auxiliary datasets with exact attention over scarce task-specific context, enabling efficient scaling to the former without compromising exact attention over the latter. We demonstrate the benefits of our approach on assays from ChEMBL and BindingDB, improving few-shot bioactivity prediction and downstream compound prioritization in retrospective Bayesian optimization.
1 INTRODUCTION
Individual bioactivity assays often lack enough measurements for reliable standalone models, motivating information sharing across assays. The introduction frames assay heterogeneity as a challenge to meta-learning and presents MetaHeta as a way to condition predictions on related-assay data.
1 INTRODUCTION
- Assay-specific prediction is few-shot because measurements are costly and often too sparse for accurate models trained in isolation.Naively pooling measurements across assays risks conflating distinct assay-specific relationships.Source: §1 · p. 1
- Meta-learning shares structure across tasks without directly pooling their measurements, using a small assay-specific context set to predict activities for new compounds.The paper treats each assay as a task, with observed compound–activity pairs as context and new compounds as targets.Source: §1 · p. 1
- Increasing meta-training task heterogeneity can degrade performance across methods and context sizes, motivating MetaHeta’s conditioning on related-assay data.The framework aims to use a more informative prior relevant to the target assay than the prior induced by the full heterogeneous training distribution.Source: §1 · p. 2
- MetaHeta uses exact attention over scarce task-specific context and linear attention over potentially large auxiliary datasets for efficient conditioning.Assay relatedness can be based on target homology or known experimental conditions.Source: §1 · p. 2
- MetaHeta improves few-shot bioactivity prediction and retrospective Bayesian optimization on assays from real-world databases.The introduction identifies ChEMBL and BindingDB as the evaluation sources.Source: §1 · p. 2
2 RELATED WORK
Related work spans gradient-based adaptation, shared representations with task-specific inference, and amortized prediction from task context. Transformer Neural Processes and Prior-Data Fitted Networks extend the context-to-prediction approach using attention and broad synthetic pretraining.
2 RELATED WORK
- Gradient-based meta-learning adapts a shared initialization, while related approaches learn shared representations for task-specific Gaussian Processes.The section also places these approaches alongside metric- and adaptation-based methods used in few-shot molecular property prediction.Source: §2 · p. 2
- Neural Processes amortize adaptation by mapping context points to predictive distributions, and Transformer Neural Processes use attention to model context and target interactions.The section describes Prior-Data Fitted Networks as pre-trained Transformer Neural Processes for off-the-shelf prediction on new tasks.Source: §2 · p. 2
- Prior-Data Fitted Networks pre-train Transformer Neural Processes on broad synthetic task distributions, a paradigm influential in tabular learning and recently applied to molecular property prediction.The authors also note that methods explicitly accounting for task heterogeneity are discussed in Appendix D.Source: §2 · p. 3
3 SETUP
The setup models each assay as a task with a finite dataset of molecular representations and assay outcomes. The goal is to predict target outcomes from an assay-specific context set and target inputs.
3 SETUP
- Each assay is modeled as an unknown function from molecular representations to outcomes, with observations generated from that function plus Gaussian noise.The setup treats the assay function as drawn from an underlying stochastic process.Source: §3 · p. 3
- An assay dataset is partitioned into context and target sets, and the meta-dataset comprises assay-specific datasets that may contain few observations.The setup denotes the context and target sizes by Nc and Nt, respectively.Source: §3 · p. 3
- The objective is to meta-learn a predictor mapping an assay-specific context set and target inputs to a predictive distribution over target outcomes.The predictor is parameterized as pθ(Y T | XT, DC).Source: §3 · p. 3
4 METAHETA
Section 4 explains how assay heterogeneity can make a shared prior too broad for few-shot adaptation, then introduces MetaHeta, which conditions predictions on data from related assays. It describes target- and description-based ways to identify related assays and an architecture that treats auxiliary data and task-specific context differently.
4.1 BEYOND A SINGLE SHARED PRIOR
- A heterogeneous meta-dataset can induce a prior too broad for limited task-specific context to concentrate the posterior on the relevant assay regime.The prior reflects the distribution of assay functions represented in the meta-dataset.Source: §4.1 · p. 3
- MetaHeta conditions predictions on auxiliary datasets from the same latent assay regime, helping approximate that regime’s posterior predictive when task-specific context is scarce.The regimes may reflect shared characteristics such as biological target or assay conditions.Source: §4.1 · pp. 3–4
4.2 DEFINING ASSAY RELATEDNESS
- Target-based retrieval uses assays of the same target, or—under a hierarchical target structure—homologous targets that share a family-level component.Same-target retrieval can be overly restrictive for understudied proteins with scarce assay data.Source: §4.2 · p. 4
- When target identities or biological relationships are unavailable, description-based retrieval ranks assays by cosine similarity between embeddings of their natural-language descriptions and applies threshold γ.The descriptions may be noisy or incomplete, and γ controls retrieval selectivity.Source: §4.2 · p. 4
4.3 MODEL
- MetaHeta shares attention parameters across coupled branches, using exact self-attention for context and FAVOR+ for auxiliary data to scale auxiliary processing while preserving exact context attention.The auxiliary branch is skipped when Daux = ∅; task-of-origin indicators distinguish observations from concatenated auxiliary datasets.Source: §4.3 · pp. 4–5, Figure 2 · p. 5
5 EXPERIMENTS
The experiments test how assay heterogeneity affects few-shot meta-learning, evaluate MetaHeta on real-world assays and retrospective compound prioritization, and examine auxiliary conditioning’s performance and cost. Across these settings, MetaHeta performs strongly, while the results show that task relatedness and context availability matter.
5 EXPERIMENTS
- The experiments use a semi-synthetic kinase benchmark, FS-Mol assays from ChEMBL, and eight BindingDB assays; results are averaged over five random seeds.The studies cover controlled heterogeneity, real-world few-shot prediction, retrospective Bayesian optimization, and auxiliary-conditioning ablations.Source: §5 · p. 5, §5.1 · p. 5, §5.2 · p. 7
5.1 META-LEARNING UNDER CONTROLLED ASSAY HETEROGENEITY
- Increasing meta-training task heterogeneity generally deteriorates performance, and broader task augmentation can impair meta-learning despite adding training data.Differences between heterogeneity regimes become more pronounced as context size increases.Source: Figure 3 · p. 6, §5.1 · pp. 6–7
- Same-subfamily augmentation improves FR2os in 25 cases, while augmentation with other tyrosine and non-tyrosine kinases decreases it in 24 and 26 cases, respectively.Outside-group augmentation performs worse than within-group augmentation in 18 comparisons.Source: §5.1 · pp. 6–7
- MetaHeta matches or exceeds all baselines across both metrics and all context sizes in the controlled benchmark.It remains advantaged even when trained with the most heterogeneous regime and compared with baselines trained in the most favorable regime.Source: §5.1 · pp. 6–7
5.2 META-LEARNING EVALUATION ON REAL-WORLD BIOACTIVITY ASSAYS
- MetaHeta ranks best in 7 of 8 FS-Mol metric–context-size comparisons and outperforms Frequent Hitters across all context sizes.Its gains are largest with the least task-specific context and diminish as more context becomes available.Source: Table 2 · p. 7, §5.2 · p. 8
5.1 META-LEARNING UNDER CONTROLLED ASSAY HETEROGENEITY
- On EPHB test tasks, MetaHeta matches or exceeds all baselines across metrics and context sizes using auxiliary information from homologous EPHA tasks.The gains are particularly large in the lowest-context regime.Source: §5.1 · p. 7
5.3 RETROSPECTIVE BAYESIAN OPTIMIZATION
- MetaHeta retrieves compounds among the 5% most active in every one of eight BindingDB assays; it finds the most active compounds in three and ranks second in three.Its average rank is 1.875, compared with 4.0 for TabPFN; the task-specific GP ranks last in four assays.Source: Figure 4 · p. 8, §5.3 · pp. 8–9
5.4 ABLATION
- One auxiliary dataset substantially improves R2os, while adding more yields little further change; five auxiliary datasets are processed faster than ICICL-TNP processes one.Expanding Daux from one to five datasets adds around 20% overhead, with each additional dataset adding about three seconds.Source: Figure 5 · p. 9, §5.4 · p. 9
6 CONCLUSION
The study finds that increasing meta-training task heterogeneity degrades few-shot performance, and introduces MetaHeta to condition predictions on data from related assays. It reports benefits across several relatedness definitions and on ChEMBL and BindingDB assays, including retrospective Bayesian optimization.
6 CONCLUSION
- Increasing meta-training task heterogeneity can degrade meta-learning performance across methods and context sizes.Source: §6 · pp. 9–10
- MetaHeta conditions predictions on related-assay data and combines scalable auxiliary-data processing with exact attention over scarce task-specific context.Source: §6 · p. 10
- MetaHeta is effective under shared-target, target-phylogeny, and assay-description relatedness, improving few-shot prediction and compound prioritization on ChEMBL and BindingDB assays.Source: §6 · p. 10
A.1 CONTROLLED-HETEROGENEITY BENCHMARK CONSTRUCTION
The controlled benchmark generates synthetic kinase assays from kinase-specific Gaussian-process posteriors across regimes of increasing task heterogeneity. The retrospective Bayesian optimization data are curated from BindingDB and selected to support compound optimization.
A.1 CONTROLLED-HETEROGENEITY BENCHMARK CONSTRUCTION
- Synthetic assays draw molecules and kinase-specific Gaussian-process activity functions, with task sizes sampled from a PERT distribution and regimes varying the source kinase group.The benchmark includes 5,000 core EPHA tasks and additional 5,000-task regimes; validation and test assays are generated from EPHA or, for homologous-target transfer, EPHB kinases.Source: §A.1 · p. 16
- The kinase regimes’ relatedness is characterized by activity-profile similarity and sequence divergence at 29 active-site residues.The activity measure is pairwise Pearson correlation over the PKIS2 panel, and the sequence measure is normalized Hamming distance.Source: §A.1 · p. 16, Table 3 · p. 17
A.2 RETROSPECTIVE BAYESIAN OPTIMIZATION
- Retrospective optimization uses eight patent-derived BindingDB assays selected for compound count, chemical diversity, non-degenerate activity, and multiple low-activity starting scaffolds.The assay descriptions summarize experimental targets, procedures, conditions, and readouts from the corresponding patent documents.Source: §A.2 · pp. 17–18
A.3 METRICS
The appendix defines evaluation metrics and describes evaluation protocols, model training, auxiliary-data construction, baselines, and computational resources. Performance is compared with an oracle for synthetic assays where the data-generating Gaussian processes are known.
A.3 METRICS
- For synthetic assays, an exact Gaussian-process oracle conditions the known generating process on each method’s context set; a value of 1 on the defined relative metrics corresponds to oracle-level performance.FNLL is the ratio of a method’s geometric mean predictive density to the oracle’s.Source: §A.3 · p. 19
A.4 IMPLEMENTATION DETAILS
- Evaluation uses context sizes 4, 8, 16, and 32, with separate fixed-context models and averaged metrics across ten context–target splits per assay.Controlled experiments share the GP molecular representation; FS-Mol and retrospective optimization retain method-appropriate representations and protocols.Source: §A.4 · pp. 19–20
A.4.1 METAHETA
- MetaHeta uses up to five auxiliary datasets, subsampled to at most 512 observations, with retrieval adapted to same-target, homologous-target, or assay-description relatedness.For FS-Mol and retrospective optimization, auxiliary datasets are selected using assay-description similarity.Source: §A.4.1 · p. 20
A.4.2 BASELINES
- Baselines include neural-process, gradient-based, kernel, pooled-data, similarity-search, random-forest, and tabular foundation-model approaches.The TNP baseline is obtained by setting MetaHeta’s auxiliary-dataset count to zero.Source: §A.4.2 · pp. 20–21
A.5 COMPUTATIONAL RESOURCES
- A MetaHeta training run took approximately 24 hours on an NVIDIA H100 GPU.Source: §A.5 · p. 21
B.1 CONTROLLED-HETEROGENEITY EXPERIMENT
The controlled experiment reports generally declining performance as training-task heterogeneity increases, while MetaHeta remains the best-performing approach across regimes. The retrospective Bayesian optimization table summarizes average method ranks across eight assays.
B.1 CONTROLLED-HETEROGENEITY EXPERIMENT
- Performance generally deteriorates as meta-training tasks become more heterogeneous, while MetaHeta remains the best-performing approach across training regimes.Source: Figure 6 · p. 22, §B.1 · p. 22
B.2 RETROSPECTIVE BAYESIAN OPTIMIZATION
- The retrospective Bayesian optimization table ranks methods by final best-observed activity averaged over five seeds, then averages ranks across eight assays.Source: Table 4 · p. 22
B.3 ABLATION
The ablation figure reports MetaHeta and ICICL-TNP NLL across auxiliary-dataset cardinalities, while the algorithm selects auxiliary assays using description similarity. The related-work discussion contrasts task-regime approaches and MetaHeta’s direct information exchange across auxiliary observations.
B.3 ABLATION
- Figure 7 compares MetaHeta and ICICL-TNP NLL across varying auxiliary-dataset cardinalities.The supplied caption identifies the comparison but does not provide its outcomes.Source: Figure 7 · p. 22
C ALGORITHMS
- The meta-training procedure constructs auxiliary data using assay-description similarity, computed from embeddings and pairwise cosine similarity.The algorithm passage notes that limits, ordering, and subsampling are omitted from its outline.Source: C ALGORITHMS · p. 23
D EXTENDED RELATED WORK
- Prior approaches either model multiple task regimes, often with a prespecified cluster count, or infer task-specific representations from context without enumerating regimes.Wang et al. (2022) represent uncertain task identity as a stochastic latent variable.Source: D EXTENDED RELATED WORK · p. 23
- MetaHeta concatenates auxiliary observations for linear self-attention, enabling direct exchange across datasets, unlike ICICL-TNP’s indirect exchange through pseudo-tokens.One-hot task indicators preserve dataset identity, but fix the maximum auxiliary-dataset count unless the architecture is modified.Source: D EXTENDED RELATED WORK · pp. 23–24
E.1 LIMITATIONS
The theoretical results do not directly guarantee gains for finite-capacity models trained on finite meta-datasets, and description-based relatedness is heuristic. The section also notes a fixed auxiliary-dataset capacity and proposes broader relatedness structures and data modalities.
E.1 LIMITATIONS
- Exact-posterior theoretical results do not directly guarantee improvements for finite-capacity models trained on finite meta-datasets.Description-based relatedness is a heuristic, although auxiliary conditioning is effective in the three tested settings; shuffling retrieved assays substantially degrades performance.Source: §E.1 · p. 24
- One-hot dataset indicators fix MetaHeta’s maximum auxiliary-dataset count, which cannot be increased after training without modifying the model.The authors report that this limitation is mild in practice when training with a conservative upper bound and evaluating with fewer datasets.Source: §E.1 · p. 24
E.2 FUTURE WORK
- Future work could model multiple levels of relatedness and incorporate protein sequence, structure, or richer assay metadata.The proposed extensions address richer biological structure and more comprehensive definitions of relatedness.Source: §E.2 · p. 24
F ATTENTION OPERATORS
- FAVOR+ uses a positive random feature map with orthogonal random features to approximate attention, and the section defines its multi-head operation.The feature map is applied rowwise to form feature matrices.Source: F ATTENTION OPERATORS · p. 25
G PROOFS
The proofs establish permutation invariance for predictions under fixed FAVOR+ features and derive a proposition under a hierarchical data-generating process. The proposition’s assumptions include conditional independence of auxiliary data from the outcome given the target’s protein-specific component.
G.1 PERMUTATION INVARIANCE
- With fixed FAVOR+ features, predictions are invariant to datapoint order within each auxiliary dataset and to auxiliary-dataset order.The proof uses permutation-equivariant attention and canonical ordering that preserves dataset indicators.Source: §G.1 · p. 25
G.2 PROPOSITION 1
- The proposition assumes auxiliary data from target homologs add no predictive information once the target’s protein-specific component is known.Its proof integrates out variables absent from the relevant integrands and uses conditional independence; conditioning does not increase entropy in expectation.Source: §G.2 · pp. 26–27