PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design

Gyubok Lee, Kiwoong Yoo, Jimin Seo, Kyunghoon Hur, Edward Choi

arXiv:2608.20755v1cs.AI

TL;DR

Wet-lab capacity limits make shortlisting generated protein binders a central bottleneck. This paper frames shortlisting as policy synthesis over precomputed proxy scores and finds modestly improved held-out Recall@10 over the strongest single-feature baseline.

  • Problem

    Limited experimental validation capacity makes selecting a small number of validated binders from generated candidate pools an unresolved shortlisting problem.

  • Method

    The paper treats post-generation shortlisting as policy synthesis, comparing fixed, supervised, and LLM-generated ranking rules over shared precomputed proxy-score panels.

  • Results

    0.589 Recall@10 modestly improves over Protenix binder ipTM’s 0.571 Recall@10 on the 10-target held-out split.

  • Takeaways & Limitations

    LLM-generated ranking policies can serve as an interpretable post-generation decision layer that combines heterogeneous proxy metrics while retaining predictor-native single-score rankings as baselines.

  • Takeaways & Limitations

    Feature-family ablations remove terms without re-optimizing the remaining policy, so they indicate policy dependence rather than monotonic feature importance.

Abstract

from arXiv · show

Modern de novo design workflows generate many candidate protein binders, but wet-lab validation capacity remains limited, making shortlisting a major bottleneck. We study whether LLMs can generate multi-metric ranking policies from precomputed structural-confidence and interface-quality proxy scores. Rather than proposing a new protein binder design pipeline, we focus on post-generation binder shortlisting: selecting the final top-K candidates from already generated binder pools using a shared panel of precomputed proxy scores. On the 10-target held-out split, averaging performance over five sampled global iterative gpt-4o policies reaches 0.589 Recall@10, modestly improving over the strongest single-feature fixed baseline, Protenix binder ipTM, which reaches 0.571 Recall@10. On the 3-target held-out subset comprising Nipah, RBX1, and TREM2, target-conditioned iterative gpt-5.4 policies reach the strongest LLM performance, with 0.519 Recall@10 and 0.583 NDCG@10. These results suggest that LLM-generated ranking policies can act as an interpretable post-generation decision layer for combining heterogeneous proxy metrics to prioritize binders from large candidate pools.

1. Introduction

The paper frames binder shortlisting as a post-generation decision problem: selecting top-K candidates from fixed pools using a shared panel of structural-confidence and interface-quality proxies. It evaluates fixed heuristics, supervised models, global LLM policies, and target-conditioned LLM policies as ranking strategies for this bottleneck.

  • Limited experimental validation capacity makes shortlisting central to converting large generated binder pools into validated hits.
  • Existing workflows commonly use fixed thresholds, single-score rankings, or hand-tuned combinations of structure-confidence and interface proxies.Examples include BindCraft’s AF2/Rosetta filters, Adaptyv’s Boltz-2 ipSAE selection, and PXDesign’s Protenix-based filters.
  • The study fixes each target’s generated candidate pool and applies a common 17-feature proxy panel before final candidate selection.The panel combines AF2-Multimer, Boltz-2, and Protenix confidence scores with interface-level descriptors from predicted complexes.
  • The controlled comparison evaluates fixed single-feature heuristics, supervised ML baselines, global LLM policies, and target-conditioned LLM policies on shared held-out pools and top-K metrics.
  • Iterative LLM policies synthesize interpretable feature-weighted combinations and are competitive with strong single-feature and supervised baselines in top-K recall.In the best settings, these policies modestly improve Recall@10.

2. Related Work

Prior binder-design work combines generation, sequence design, and structure-based validation, with candidate selection typically handled by workflow-specific filters or ranking rules. Related methods address target-aware design, binding propensity, or agentic assistance, whereas this work synthesizes a separate policy for selecting K candidates from generated pools using precomputed proxy scores.

  • Binder pipelines and candidate selection: Modern workflows combine RFdiffusion backbone generation, ProteinMPNN-style sequence design, and structure-based validation or filtering with predictors such as AF2.
  • Binder pipelines and candidate selection: Existing systems select candidates using workflow-specific rules, including fixed AF2-confidence, Rosetta, and interface-quality filters or Boltz-2 ipSAE ranking with voting and expert curation.These examples include BindCraft and Adaptyv’s Nipah release.
  • Target-aware binder design: Target-aware design methods generate binders from target structure alone or learn surface fingerprints to parameterize interaction design.
  • Binder pipelines and candidate selection: APPRAISE ranks engineered proteins by target-binding propensity through structure modeling, while these prior works focus on generation or pairwise compatibility scoring.
  • Binder pipelines and candidate selection: This work instead takes a generated pool with precomputed proxy scores and synthesizes a separate decision policy for selecting K candidates.
  • LLMs and agents for protein design: ProtAgents and ProteinCrow use LLM-backed or agentic systems to retrieve knowledge, analyze structures, call tools, and incorporate structural, literature, or biochemical context.

3. Problem Setup

The problem is to synthesize ranking policies that shortlist K candidates from each generated binder pool using fixed proxy-score vectors, maximizing recall of experimentally verified binders under a validation budget. A policy selects features, assigns weights and directions, then ranks candidates by an aggregate normalized score.

  • Shortlisting as policy synthesis: For target t, the task selects a subset S_t of size K from candidate pool C_t to maximize recall of experimentally verified binders under the validation budget.Each candidate has a precomputed proxy-score vector x_t,c shared across the pool’s common features.
  • Shortlisting as policy synthesis: Policy synthesis determines which proxy scores to combine and how strongly to weight them for the target pool.
  • Policy space: A policy is the structured triple (F, w, g), with F selecting features and w assigning integer weights from {1, 2, 3}^{|F|}.
  • Policy space: The main LLM policy normalizes selected features within each target pool, computes a weighted aggregate, sorts candidates descending, and returns the top K.Each selected feature also has a predefined higher-is-better or lower-is-better direction.

4. Datasets

The study assembles candidate-level binder-design datasets into target-disjoint retrospective shortlisting episodes, with fixed candidate pools and binder/non-binder labels. Each candidate is represented by a 17-feature proxy-score panel derived from predicted complexes and confidence outputs.

  • Dataset construction: Eight public sources or workflow releases provide candidate-level experimental outcomes, defining fixed retrospective shortlisting episodes with binder/non-binder labels.A source is included only when it reports outcomes for tested designs.
  • Dataset splits: The development split contains 11 BoltzGen targets, while the held-out split contains 10 target-disjoint targets from independent workflow outputs and public validation releases.Held-out sources include BindCraft revalidation, pMHC minibinders, Nipah, RBX1, and TREM2.
  • Preprocessing: Merged releases for the same biological target are pooled and exact candidate amino-acid sequences are de-duplicated; EGFR yields 605 unique designs.Preprocessing also covers binding-outcome parsing, source-specific target assignment, and zero-positive target handling.
  • Proxy features: Each candidate has 17 active proxy scores, including Boltz-2 pDockQ2 and ipSAE interface proxies and Rosetta InterfaceAnalyzer geometry, burial, and energy proxies.Feature definitions, monotonic directions, and extraction details are provided in Appendix C.
  • Inference settings: For each target, one target-chain MSA is precomputed and reused across binders, while de novo binder chains remain single-sequence during AF2-Multimer, Boltz-2, and Protenix/PXDesign prediction.AF2-Multimer uses 3 models and 3 recycles; Boltz-2 uses 5 diffusion samples, 3 recycling steps, and 200 sampling steps.

5. Experimental Setup

The study compares fixed heuristics, supervised baselines, and sampled LLM ranking policies under global and target-conditioned evaluation protocols. It fixes K = 10 and evaluates Recall@10, Precision@10, and NDCG@10, with policy-dependence checks through proxy-score family ablations.

  • Evaluation settings: Global methods use one rule across held-out targets, whereas target-conditioned methods generate a separate rule for each target.Target-conditioned methods may select different feature subsets or weights for different candidate pools.
  • Baselines: The baselines include single-score structure-prediction heuristics, logistic regression, and XGBoost using the shared 17-feature panel.Supervised models test whether direct learning over the feature panel is sufficient without target-conditioned policy synthesis.
  • LLM policies: LLM policies are evaluated with gpt-4o across the 10-target and 3-target held-out splits and with gpt-5.4 on the 3-target subset.Global prompts use development calibration context, while target-conditioned prompts additionally use unlabeled held-out pool statistics at test time.
  • LLM policies: Each sampled LLM policy selects 3 to 5 features with positive integer weights in {1, 2, 3}, and reported results average five independently generated policies.Features are normalized within each target pool, direction-corrected, and combined into a weighted rank score.
  • Ablations: Proxy-score family ablations remove selected feature families without regenerating or re-optimizing policies, so they test policy dependence rather than monotonic feature importance.The remaining weights stay fixed and the same deterministic executor is re-evaluated.
  • Metrics: K = 10 is fixed before evaluation, with Recall@10 as the primary metric and Precision@10 and NDCG@10 as secondary metrics.Recall@10 uses denominator min(K, nbinders), Precision@10 measures wet-lab hit rate, and NDCG@10 measures binder concentration near the ranking top.

6. Results

The results show that averaged global iterative gpt-4o policies modestly outperform the strongest fixed baseline on the 10-target split, while target-conditioned iterative gpt-5.4 achieves the strongest LLM performance on the 3-target subset. Generated policies concentrate on a small set of structure-confidence and interface-quality proxy scores.

  • 10-target held-out split: 0.589 Recall@10 and 0.404 Hit@10 were achieved by averaging five global iterative gpt-4o policies on the 10-target held-out split.The strongest single-feature fixed baseline, Protenix binder ipTM, reached Recall@10 = 0.571 and Hit@10 = 0.360.
  • Evaluation setup: Table 3 reports held-out evaluation at K = 10 for both the 10-target split and the 3-target subset, with LLM results averaged over five sampled policies.Recall@10 uses min(K, nbinders) as its denominator, and Hit@10 is equivalent to Precision@10.
  • 3-target held-out subset: 0.519 Recall@10, 0.513 Hit@10, and 0.583 NDCG@10 were achieved by target-conditioned iterative gpt-5.4 on the Nipah, RBX1, and TREM2 subset.This was the strongest LLM performance; Protenix binder ipTM and LR-BG without regularization were the strongest fixed and supervised baselines.
  • Generated LLM policy composition: Target-conditioned iterative gpt-4o policies concentrate weight on a small set of structure-confidence and interface-quality scores rather than distributing weight uniformly.Across five policy samples for each of 10 held-out targets, every policy selected Boltz-2 ipTM and Protenix pair ipTM; Rosetta shape complementarity appeared in 40 of 50 policies.

7. Analysis and Discussion

Iterative feedback substantially improves global LLM ranking, while target conditioning provides selective gains rather than uniformly better recovery. Strong fixed single-score baselines remain competitive, but target-conditioned iterative policies perform best on the 3-target subset.

  • Global versus target-conditioned policies: 0.584 Recall@10 and 0.523 NDCG@10 show target-conditioned iterative gpt-4o improves ranking quality without uniformly surpassing global recovery.The global iterative policy reaches 0.589 Recall@10, whereas target-conditioned iterative gpt-4o reaches the highest LLM NDCG@10 on the same split.
  • Global versus target-conditioned policies: Target conditioning adapts to target-specific proxy-score distributions by adjusting which feature families to trust and how strongly to weight them.The executor remains deterministic and applies fixed directions, while the prompt can inspect score spread, skew, and relative behavior within a held-out candidate pool.
  • What iterative feedback adds: 0.589 versus 0.435 Recall@10 shows iterative feedback improves global gpt-4o over the single global policy on the 10-target held-out split.Iterative prompting calibrates policies using development-search feedback about accepted or rejected feature-weighted rules and their aggregate metrics.
  • Global versus target-conditioned policies: 0.519 versus 0.502 Recall@10 and 0.583 versus 0.514 NDCG@10 show target-conditioned iterative gpt-5.4 improves over single-turn target-conditioned gpt-5.4 on the 3-target subset.The subset comprises Nipah, RBX1, and TREM2 as described in the paper context.
  • Strong single-score baselines remain important: 0.571 Recall@10 makes Protenix binder ipTM the strongest fixed one-feature baseline on the 10-target held-out split.It reaches 0.504 Recall@10 on the 3-target held-out subset and remains an important comparison for generated policies.

8. Conclusion

The paper frames binder shortlisting as post-generation policy synthesis over fixed candidate pools and precomputed proxy scores. LLM-generated policies provide interpretable, target-adjustable rules for combining heterogeneous signals while retaining predictor-native rankings as explicit baselines.

  • Conclusion: The study selects final top-K binders from fixed generated pools using precomputed structural-confidence and interface-quality proxy scores.It compares fixed heuristics, supervised baselines, and LLM-generated ranking policies.
  • Conclusion: LLM-generated policies combine AF2, Boltz-2, Rosetta interface, and occasional Protenix confidence signals into interpretable feature-weighted ranking rules.The rules can be adjusted to each target pool.
  • Conclusion: LLM-generated ranking policies can serve as an interpretable post-generation decision layer for prioritizing binders from heterogeneous candidate pools.The conclusion positions this layer alongside, rather than instead of, structure-prediction and interface-quality metrics.
  • Conclusion: Strong predictor-native single-score rankings should remain explicit baselines when evaluating LLM-generated policies.The conclusion emphasizes that generated policies do not replace structure-prediction and interface-quality metrics.

A. Dataset Preprocessing

The dataset is preprocessed by retaining experimentally labeled candidate pools, assigning candidates to assay-defined targets, and excluding cases that cannot support recall-based evaluation. Multiple releases for the same biological target are merged with exact-sequence deduplication, while source-level pools remain available for audits and feature checks.

  • Candidate inclusion: Candidate pools retain designs only when candidate sequence, target sequence, and candidate-level experimental binding outcome are publicly available.Entries lacking explicit binding measurements are treated as unlabeled and excluded from recall-based evaluation; this removes 7 Adaptyv EGFR round-1 submissions.
  • Target assignment: Target identifiers follow the experimental assay target, excluding off-target or control records from intended-target labels.Single-target competitions use the competition target, while BindCraft1 revalidation candidates are assigned by assay target; BoltzGen structural seed identifiers are mapped to released biological assay targets.
  • Evaluation eligibility: Targets with zero positives—GM2A, ORM2, and TNF-α—are excluded because Recall@10 is undefined when the target-level positive denominator is zero.Source-level pools, including BoltzGen targets without released positives, are retained for audit and feature-extraction checks.
  • Pool consolidation: Multiple releases containing the same biological target are merged into one evaluation pool and exact duplicate candidate amino-acid sequences are removed.The preprocessing retains source-level pools separately for audit and feature-extraction checks.

B. Temporal Leakage Audit

The temporal-leakage audit documents model cutoffs and conservative public-availability dates for candidate-level labels and source pools. Both held-out splits use labels released after gpt-4o’s cutoff, while Nipah, RBX1, and TREM2 postdate gpt-5.4’s cutoff.

  • Model cutoffs: gpt-4o-2024-11-20 uses a documented cutoff of 2023-10-01, after which labels for both held-out splits were released.The audit includes development entries for provenance and auditability, but they are not held-out evaluation labels.
  • Model cutoffs: 2025-08-31 is the gpt-5.4 cutoff, and Nipah, RBX1, and TREM2 became public after that date.These three targets are the held-out targets used for the gpt-5.4 comparison.
  • Dataset-level audit: The audit uses conservative public-availability dates for candidate-level experimental labels or stable public data-package availability when explicitly noted.For ProteinBase, overview “Results released” dates are preferred over potentially stale stage-detail text; BindCraft1 revalidation is public.

C. Feature Glossary … F. Per-target Performance

The paper defines a reproducible 17-feature proxy panel, controlled prompt conditions, and structured deterministic ranking policies for evaluating global and target-conditioned shortlisting across held-out targets. It also documents the Cao/Bennett transfer baseline and per-target evaluation setups for gpt-4o and gpt-5.4.

  • C. Feature Glossary: The main policy panel contains 17 reproducible candidate-level features with fixed monotonic directions and extraction summaries.The panel combines target-MSA Boltz-2, AF2-Multimer, Protenix, and Rosetta complex outputs; single-sequence predictions are retained only for ablation.
  • C. Feature Glossary: Target-MSA complex predictions use the target-chain MSA while designed binder chains remain single-sequence.Target-chain A3M files are generated once with the Protenix/PXDesign-compatible MMseqs2 MSA service and cached before feature extraction.
  • D. Cao/Bennett AF2 Transfer Baseline: The historical Cao/Bennett transfer baseline uses AF2 binder-target interface PAE and binder-chain pLDDT measurements, recomputed on current held-out pools with the AF2-Multimer target-MSA protocol.These features correspond to compatible confidence measurements available in both historical training pools and current held-out pools.
  • E. Prompt Information Conditions: Prompts provide feature descriptions, fixed directions, development-set single-feature metrics, and per-development-target score-distribution summaries, while target-conditioned prompts add target-specific information.Global and target-conditioned policies use the same feature panel, policy class, deterministic executor, and held-out-label restriction.
  • E. Prompt Information Conditions: Each valid policy selects 3 to 5 features with integer weights in {1, 2, 3}, and the deterministic executor applies fixed directions, normalization, aggregation, and constraints.Prompts prohibit hard thresholds, feature-specific filters, direct-affinity interpretation of proxies, and overriding feature directions or aggregation.
  • E. Prompt Information Conditions: Iterative variants add aggregate development-search feedback about evaluated feature-weight combinations and summary metrics without exposing held-out labels.The feedback is computed on the development split and preserves the single-turn information restrictions.
  • F. Per-target Performance: Per-target evaluation reports gpt-4o Recall@10 on the 10-target held-out split and gpt-5.4 detailed metrics on the 3-target Nipah, RBX1, and TREM2 subset.Table 10 compares Protenix binder ipTM with matched global, target-conditioned, and iterative gpt-4o policies; Table 11 reports the corresponding gpt-5.4 settings.
Loading 2608.20755v1…