PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings

Kevin Tirta Wijaya, Alston Lo, Michael Sun, Wojciech Matusik, Vahid Babaei

arXiv:2610.01827v1cs.LG

TL;DR

Scientific design candidates are abundant, but physical validation capacity is scarce, and design discovery is often comparative. PRISMS aggregates pairwise rankings from multiple rankers and selectively queries higher-fidelity sources. Across screening and optimization, it improves discovery rate and extent, including ∼18.8% higher hypervolume than GP-BO.

  • Problem

    Candidate designs are abundant while physical validation capacity is scarce, and scientific design discovery is often comparative rather than focused on exact property values.

    Source: §1 · pp. 1–2

  • Method

    PRISMS aggregates pairwise comparisons across rankers and selectively queries higher-fidelity rankers to guide curation.

    Source: §1 · p. 2

  • Results

    Across screening and optimization, PRISMS improves discovery rate and extent; it achieves ∼18.8% higher hypervolume than GP-BO in optimization.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Multi-fidelity pairwise rankings can guide curation when experimental measurements are scarce and accelerate discovery by reducing required experimental rounds.

    Source: §1 · p. 3, §5 · p. 10

    PRISMS assumes access to rankers ordered by fidelity and cost, whose judgments may nevertheless be noisy and mutually inconsistent.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a validation congestion where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on curation: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ~18.8% higher hypervolume than the Bayesian optimization baseline.

1 INTRODUCTION

Scientific design candidates are increasingly abundant, but slow, costly physical validation makes selecting which designs to test a central curation challenge. PRISMS uses multi-fidelity pairwise rankings to guide that selection and improve discovery across screening and optimization.

1 INTRODUCTION

  • Scientific design discovery faces validation congestion: candidate designs are abundant, but physical validation is slow and costly, making resource allocation central to curation.A drug-discovery experimental round typically takes one to three months.Source: §1 · p. 1
  • Regression-based curation depends on experimental labels, which are costly to acquire under validation congestion and may be insufficient for reliable models.Regression can be effective when sufficient training labels are available.Source: §1 · pp. 1–2
  • Pairwise rankings provide comparative curation signals without requiring exact property values and can be drawn from rankers with different fidelity and cost.The passage describes rankings as often easier to elicit reliably than absolute property estimates.Source: §1 · p. 2
  • PRISMS aggregates comparisons in a graph, estimates unmeasured designs with a regularized Bradley-Terry model, and escalates queries to higher-fidelity rankers using a Fisher-information criterion.Higher-fidelity queries focus on comparisons that can improve property estimates most, and the resulting estimates guide experimental validation.Source: §1 · p. 2
  • PRISMS reaches 50% top-10 discovery recall in ∼42% fewer iterations than conventional pool-based active learning, and achieves ∼18.8% higher hypervolume than GP-BO.The evaluations cover fixed-library screening and optimization without a predefined library; PRISMS also reduces the number of experimental rounds needed for discovery.Source: §1 · pp. 2–3

2 BACKGROUND

This background frames scientific design discovery as finding new designs that meet an objective and reviews surrogate-model and pairwise-ranking approaches to curation. It distinguishes absolute-value supervision from comparative judgments that can also refine estimates on an absolute scale.

2 BACKGROUND

  • Scientific design discovery concerns identifying new drugs, materials, or engineered objects that satisfy an objective, with curation deciding which designs to validate.The paper distinguishes this focus from discovering relationships, mechanisms, or hypotheses about existing systems.Source: §2 · p. 3
  • Surrogate models, including data-driven models and physics-based models such as molecular docking, can guide validation but rely on absolute property measurements that may be scarce.Gaussian Processes can be paired with UCB or EI to choose designs for validation.Source: §2 · p. 3
  • Pairwise-ranking methods recover scores, top-k partitions, or global orderings; some also combine rankings with experimental values to refine absolute-scale estimates.The passage cites Spectral MLE, HodgeRank, active comparison selection, LLM rankers, Projection, and RankRefine++.Source: §2 · p. 3

3 DESIGN CURATION WITH MULTI-FIDELITY PAIRWISE RANKINGS

PRISMS curates designs by combining experimental measurements with pairwise rankings from rankers of increasing fidelity. It jointly estimates unmeasured designs, selectively escalates comparisons, and sends the highest-estimated designs for validation.

3 DESIGN CURATION WITH MULTI-FIDELITY PAIRWISE RANKINGS

  • PRISMS selects a budget-limited batch of unmeasured designs for experimental validation in each campaign round, then uses the new measurements in subsequent rounds.Campaigns use fixed libraries for screening or generate designs during optimization.Source: §3 · p. 3

3.1 JOINT PROPERTY ESTIMATION FROM PAIRWISE RANKINGS

  • The rank-outcome graph connects each unmeasured design to validated designs and nearby unmeasured designs, with edges defining which pairs are eligible for ranking.Connections are recomputed at the start of each round; within a round, they remain fixed.Source: §3.1 · p. 4
  • A regularized Bradley–Terry model jointly estimates unmeasured designs’ properties from pairwise rankings while keeping experimentally measured values fixed.The objective includes a prior estimate, and the final estimates determine which designs are selected for validation.Source: §3.1 · pp. 4–5

3.2 INFORMATION-GUIDED ESCALATION TO HIGHER-FIDELITY PAIRWISE RANKER

  • Higher-fidelity rankers have limited query budgets, so PRISMS escalates pairs with high outcome and edge uncertainty using a local Fisher-information gain score.The outcome uncertainty is greatest when estimated properties are equal; high edge uncertainty marks a poorly constrained property difference.Source: §3.2 · p. 5

3.3 RANKREFINE++ AS A SPECIAL CASE

  • RankRefine++ is a special case of PRISMS when there is one unmeasured design, edges connect only to measured designs, and only one ranker is used.Under the stated parameter setting, the objective is proportional to RankRefine++’s per-design MAP objective and yields the same estimate.Source: §3.3 · p. 6

4 EXPERIMENTAL RESULTS

Across fixed-library screening, engineering campaigns, and generated-molecule optimization, PRISMS outperforms evaluated baselines. Ablations and sensitivity tests examine its components and performance under different ranker conditions.

4.1 DRUG DISCOVERY CAMPAIGN

  • PRISMS improves AUDC@10 by 25.1% over HodgeRank and 24.3% over RankRefine++, and top-10% F1 by 31.2% over RankRefine++.Its 24-round restricted mean time to 50% discovery recall is 11.1 rounds, compared with 19.3 for GP-EI. Campaigns cover 14 drug-discovery datasets over 12 rounds; results aggregate 30 seeds.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 ENGINEERING DISCOVERY CAMPAIGN

  • Across three engineering datasets, PRISMS outperforms all evaluated baselines across all performance metrics.The campaigns use 10 random seeds per dataset and report aggregated 12-round results.Source: §4.2 · p. 7

4.3 MULTI-OBJECTIVE DRUG OPTIMIZATION CAMPAIGN

  • At the end of 12 rounds, PRISMS achieves about 14.3% higher hypervolume than HodgeRank and RankRefine++ in JNK3 activity and solubility optimization.Candidates are generated throughout the campaign, and performance is measured by normalized hypervolume of oracle-validated designs.Source: §4.3 · pp. 7–8, Figure 4 · p. 7

4.4 ABLATION STUDY

  • Removing unlabeled-node edges reduces AUDC@10 by 7.6%; removing edge or outcome uncertainty reduces it by 4.6% or 2.3%, and random escalation reduces it by 7.7%.A single fidelity stage lowers AUDC@10 by 16.5%, while adding a third stage improves it by 11.4% over the default two-stage setup.Source: §4.4 · p. 8

4.5 SENSITIVITY ANALYSIS

  • Increasing higher-fidelity query budgets improves PRISMS’ AUDC@10 across all three noise models; accuracy gains grow with budget, and PRISMS beats RankRefine++ in nearly all settings.Difficulty-dependent noise yields the strongest overall performance and largest gains over RankRefine++.Source: §4.5 · p. 8, Figure 5 · p. 9

4.6 RECOVERY OF MISSED TOP-10 DESIGNS

  • Across 14 drug-discovery datasets, PRISMS shifts true top-10 designs toward top ranks faster than competitors, while GP-EI remains close to random sampling.The heatmaps track normalized EMD over 12 rounds; lower EMDnorm@10 means true top-10 designs are nearer their target ranks.Source: §4.6 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION

PRISMS addresses validation congestion by using multi-fidelity pairwise rankings to curate designs when experimental measurements are scarce. Across screening and optimization, it improves discovery, and the authors conclude that better use of available scientific knowledge can complement acquiring more measurements.

5 CONCLUSION

  • PRISMS shifts curation from data-hungry regression toward multi-fidelity pairwise rankings when experimental measurements are scarce.It integrates rankings from diverse experts and selectively escalates to higher-fidelity rankers.Source: §5 · p. 10
  • Across fixed-library screening and optimization, PRISMS improves both the rate and extent of discovery of high-performing designs.The authors suggest that discovery need not rely solely on acquiring more measurements, but can also use available external scientific knowledge.Source: §5 · p. 10
  • Generative AI tools assisted with implementation, literature identification, and readability edits, but not with developing the study's models, hypotheses, methods, or result interpretations.The authors state that they reviewed and verified all AI-assisted work, including generated code and text edits.Source: AI USE STATEMENT · p. 10

A LIMITATIONS

The evaluation is mostly retrospective, with higher-fidelity rankers primarily simulated using noisy oracles. The authors also identify limited LLM-ranker testing and PRISMS's fixed-fidelity-ordering assumption as boundaries for interpreting the work.

A LIMITATIONS

  • Most experiments are retrospective, and higher-fidelity rankers are primarily simulated, so real campaigns may involve unrepresented distribution shifts and domain-specific biases.The authors consider the controlled setting appropriate for comparing curation methods under matched ranking access and examining sensitivity to ranker accuracy and error structure.Source: A LIMITATIONS · p. 14
  • The agentic-ranker study is exploratory: it tests one LLM-based ranker across three drug-discovery tasks and three random seeds.A systematic benchmark across multiple LLMs would provide complementary insights; in this setup, Claude Fable declined biochemical ranking queries under its safety mechanisms.Source: A LIMITATIONS · p. 14
  • PRISMS assumes a fixed fidelity ordering, although ranker quality may vary by pair, domain, and accumulated experimental evidence.For example, an expert chemist may rank familiar scaffolds more accurately, and their accuracy on unfamiliar scaffolds may improve as evidence accumulates.Source: A LIMITATIONS · p. 14

B.1 COMPUTATIONAL SCALING ANALYSIS

The scalability analysis measures PRISMS's one-round runtime and peak memory on synthetic libraries from 1,000 to 100,000 candidates. Runtime and memory rise with library size, but the reported computational overhead remains small relative to physical validation.

B.1 COMPUTATIONAL SCALING ANALYSIS

  • The analysis measured one-round wall-clock runtime and peak resident set size for two ranker-fidelity levels on synthetic libraries of 1,000 to 100,000 candidates.Runtime measurements used an Intel Xeon Sapphire Rapids 48-core/96-thread 2.10GHz system.Source: §B.1 · p. 14
  • At 100,000 candidates, PRISMS took 3,233 s and used 1.03 GB peak RSS, compared with 1.45 s and 282 MB at 1,000 candidates.At 10,000 candidates, runtime was 32.47 s and peak RSS was 356 MB.Source: §B.1 · p. 14
  • Even at 100,000 candidates, the authors judge PRISMS's computational overhead small relative to physical validation, which takes days or months.They therefore consider computation unlikely to be the dominant bottleneck in this setting.Source: §B.1 · p. 14

B.2 SENSITIVITY ANALYSIS

The sensitivity analysis finds that performance depends on the prior, ranker weights, and query-neighborhood size. In a separate agentic-ranker evaluation, PRISMS outperforms two oracle-based baselines but remains below the simulated-oracle result.

B.2 SENSITIVITY ANALYSIS

  • AUDC@10 remains relatively stable for prior regularization λ2 ≤ 10^-1, then declines to approximately 0.33 at λ2 = 100.The authors interpret this as supporting regressor predictions as weak priors that pairwise rankings can refine in label-scarce regimes.Source: §B.2 · p. 14
  • Increasing λ(1)1 beyond 10^-1 reduces AUDC@10 from approximately 0.41 to 0.32; λ(2)1 performs best near 10 before declining at 100.The results support greater, but not full, confidence in higher-fidelity rankers with higher accuracy.Source: §B.2 · p. 14
  • Smaller query-query neighborhoods kU generally yield higher AUDC@10 than larger ones, with performance approximately plateauing for kU ≥ 10.Thus, denser connectivity does not necessarily improve discovery.Source: §B.2 · p. 14

B.3 AGENTIC PRISMS WITH LLM AS HIGHER-FIDELITY RANKER

  • Agentic PRISMS achieves AUC 0.105, below 0.162 for PRISMS with a 75%-accurate simulated oracle, but above RankRefine++ at 0.093 and HodgeRank at 0.092.The authors describe this as suggesting feasibility for real-world scientific design curation.Source: §B.3 · p. 15

B.4 CHARACTERIZATION OF LLM AS A PAIRWISE RANKER

The LLM ranker is more accurate on pairs with larger standardized property gaps, while PRISMS improves AUDC@10 across drug-discovery datasets. The engineering campaign reports top-10 discovery recall and Top-10% F1 AUC across three datasets.

B.4 CHARACTERIZATION OF LLM AS A PAIRWISE RANKER

  • LLM ranking accuracy rises with property gap, reaching approximately 0.65–0.66 for the largest gaps in pooled results.Accuracy is near chance for pairs with similar property values; the analysis bins comparisons by standardized property gap across three datasets and pooled results.Source: §B.4 · pp. 15–16

B.5 DRUG DISCOVERY CAMPAIGN

  • PRISMS improves AUDC@10 over RankRefine++ and HodgeRank on all drug-discovery datasets, with statistically significant results.Figure 9 shows dataset-wise paired differences; positive values indicate higher discovery performance for PRISMS.Source: §B.5 · p. 16, Figure 9 · p. 17

B.6 ENGINEERING DISCOVERY CAMPAIGN

  • Figure 10 reports top-10 discovery recall over 12 rounds and Top-10% F1 AUC across three engineering datasets.Results are means and standard errors over 10 random seeds; the supplied passages do not state comparative outcomes.Source: Figure 10 · p. 17, §B.6 · p. 17

B.7 ABLATION STUDY

The appendix describes the PRISMS ablation scope and implementation choices for graph construction, estimation, uncertainty scoring, and a rule-based ranker.

C.1 GRAPH CONSTRUCTION

  • Each unmeasured design connects to all measured designs and its k_U nearest unmeasured neighbors by prior-estimate difference; the graph stays fixed during the round.Comparisons are stored by edge and ranker identity.Source: §C.1 · p. 17

B.7 ABLATION STUDY

  • The ablation compares PRISMS components and fidelity stages using 24-round mean AUDC across 14 drug-discovery datasets and 10 seeds.Relative change is measured against the default two-stage PRISMS configuration; the supplied passages do not provide table values.Source: §B.7 · p. 17, Table 2 · p. 18

C.2 OPTIMIZATION.

  • The method holds experimentally validated values fixed and optimizes unmeasured estimates with L-BFGS-B, initializing from priors and then the preceding fidelity stage.The implementation defaults to at most 300 iterations, with function and gradient tolerances of 10^-12 and 10^-8.Source: §C.2 · p. 18

C.3 SPARSE HESSIAN CONSTRUCTION.

  • The analytic Hessian combines comparison contributions, while the prior adds 2λ^2I and graph structure determines off-diagonal sparsity.Comparisons between unmeasured designs contribute off-diagonal entries; comparisons against validated designs contribute only to unmeasured-design diagonals.Source: §C.3 · p. 18

C.4 COMPUTING EDGE UNCERTAINTY.

  • A sparse LU factorization is reused to evaluate each eligible comparison’s uncertainty by linear solves, avoiding a dense covariance matrix.The quadratic form is evaluated through direct linear solves.Source: §C.4 · p. 18

C.5 LLM-GENERATED RULE-BASED RANKER

  • The rule-based CLint ranker scores molecular features with weighted terms, then compares the train and test feature scores.The example is generated using Claude Sonnet 4.6 and weights aromatic rings, logP, heteroatoms, rotatable bonds, and fraction Csp3.Source: §C.5 · pp. 18–19

C.6 DATASET

The section identifies the datasets used in the experiments.

C.6 DATASET

Related papers

Loading 2610.01827v1…