ProteinBench: A Holistic Evaluation of Protein Foundation Models

1 INTRODUCTION

• Existing task- and model-specific evaluation criteria make meaningful comparisons between protein foundation models and their relative strengths and limitations difficult.The paper also motivates standardized benchmarks as guidance for future research and model selection in practical applications.Source: §1 · p. 2
• ProteinBench classifies generative challenges across protein design, structure prediction, and conformational dynamics, using curated datasets to examine protein modalities.Protein design includes structure, sequence, structure-sequence co-design, and antibody design.Source: §1 · p. 2
• ProteinBench assesses model performance across quality, novelty, diversity, and robustness rather than relying on only one or two aspects.The paper notes that protein modeling and design involve interplay between sequence, structure, and function.Source: §1 · p. 3
• Analyses reflect different user objectives, including fitting natural evolutionary distributions or generating novel proteins outside the training-set distribution.The stated aim is to provide insights relevant to a range of practical applications.Source: §1 · p. 3
• The unified framework provides a public leaderboard and open-source code for benchmarking models against existing methods and contributing new evaluations.ProteinBench aims to establish standardized, comprehensive, and user-centric evaluation.Source: §1 · p. 3

2.1.1 INVERSE FOLDING

• Inverse-folding evaluations use newly released CASP15 and CAMEO structures for native-distribution objectives and RFdiffusion-generated backbones of 100–500 residues for de novo design.The section frames these as distinct user objectives in structure-based sequence design.Source: §2.1.1 · p. 4

2.1.2 PROTEIN BACKBONE DESIGN

• Backbone design evaluates generated structures for refoldability, novelty against known folds, and diversity, using 100 distinct PDB structures as distribution references.The reference set is iteratively filtered to retain structurally distinct single-chain proteins.Source: §2.1.2 · pp. 4–5

2.1.3 PROTEIN SEQUENCE DESIGN

• Sequence design evaluates naturalness with ProGen2 perplexity and predicted structural stability, diversity, and novelty, with UniRef50 identified as a commonly used training dataset.Generated sequences are folded with ESMFold for structure-based assessment.Source: §2.1.3 · p. 5

2.1.4 STRUCTURE AND SEQUENCE CO-DESIGN; 2.1.5 MOTIF SCAFFOLDING

• Co-design jointly optimizes backbone and sequence, assessing structure quality, compatibility, and novelty; motif scaffolding places a functional site in a stable framework.Motif scaffolding evaluation also considers motif accuracy and retention of function.Source: §2.1.4 · p. 5, §2.1.5 · p. 5

2.1.6 ANTIBODY DESIGN

• Antibody design targets antigen binding, often focusing on CDR regions or CDR-H3, and evaluates functionality, specificity, rationality, and sometimes similarity to natural antibodies.The section notes that SAbDab is commonly used, but its structural data are limited and redundant.Source: §2.1.6 · p. 6

2.2.1 PROTEIN FOLDING: SINGLE-STATE PREDICTION

• Single-state folding predicts a protein structure from sequence and is evaluated against PDB references using structural-similarity and structural-quality metrics.The evaluation uses 183 short-to-midlength single-chain CAMEO2022 targets.Source: §2.2.1 · p. 6

2.2.2 MULTIPLE-STATE PREDICTION

• Multiple-state prediction samples distinct observed conformations; evaluation covers ensemble accuracy, sample diversity, and structural quality on apo-holo and BPTI datasets.The apo-holo set contains 91 proteins, while BPTI conformations come from five clusters identified in a prior long-time MD simulation.Source: §2.2.2 · p. 6

2.2.3 DISTRIBUTION PREDICTION

• Distribution prediction seeks samples resembling a target conformational distribution, assessed for flexibility, distributional accuracy, and ensemble observables against MD references.The benchmark uses 82 ATLAS proteins whose PDB entries postdate May 1, 2019 and are outside the relevant training or validation sets.Source: §2.2.3 · pp. 6–7

3.1.1 INVERSE FOLDING

• Language model-based methods capture native evolutionary fitness, while ProteinMPNN performs better on de novo backbone design; neither excels across both objectives.ProteinMPNN was trained with 0.2 Å coordinate noise; ESMIF1 and ESM3 were not measured on the evolution-distribution objective because they use native structures and sequences in training.Source: §3.1.1 · p. 7, Table 2 · p. 8

3.1.2 STRUCTURE DESIGN

• RFdiffusion leads backbone quality at 50–300 residues, while all models’ scTM scores fall by more than 20% at 500 residues.FoldFlow and Genie perform well on novelty under scTM >0.5; Chroma generally leads under scTM >0.8 and shows commendable diversity. FoldFlow’s smaller training set may make comparisons unfair, particularly for longer chains.Source: §3.1.2 · p. 8, Table 3 · p. 9

3.1.3 SEQUENCE DESIGN; 3.1.4 STRUCTURE AND SEQUENCE CO-DESIGN

• DPLM leads sequence-generation quality, while EvoDiff and ESM3 favor diversity and novelty; ProGen2 offers a balance across metrics.ProteinGenerator and Multiflow show strong structure–sequence compatibility across lengths, with Multiflow particularly robust; ProtPardelle and ESM3 degrade as sequence length increases.Source: §3.1.3 · p. 9, Table 4 · p. 9, §3.1.4 · p. 10

3.1.5 MOTIF-SCAFFOLDING

• Structure-based methods generally perform well across scaffold contexts, with RFdiffusion particularly robust; ESM3 is competitive but inconsistent, and no method excels universally.Sequence-based methods vary by scaffold, with stronger performance on some evolution-governed scaffolds and weaker performance on more complex structural motifs.Source: §3.1.5 · pp. 10–11, Figure 2 · p. 11

3.1.6 ANTIBODY DESIGN

• Antibody metric leaders diverge: accuracy, binding energy, structure–sequence consistency, and specificity favor different methods.For example, dyMEAN and MEAN lead accuracy, AbDPO variants lead binding-energy results, and dyMEAN/dyMEAN-FixFR lead scRMSD; the two consistency metrics can disagree.Source: §3.1.6 · pp. 11–13
• No antibody method wins across all metrics, and all show substantial gaps from natural antibodies; AbDPO++ is among the strongest across aspects without obvious weaknesses.The authors identify limited structured data and lack of atomic-level antibody–antigen modeling as challenges.Source: Table 6 · p. 12, §3.1.6 · p. 13

3.2.1 PROTEIN FOLDING: SINGLE-STATE PREDICTION; 3.2.2 MULTIPLE-STATE PREDICTION

• AlphaFold2 and OpenFold lead single-state accuracy; for BPTI, ConfDiff with force guidance gives the best RMSDens, but apo–holo changes remain difficult to sample.Increasing sample depth improves expected accuracy for most multiple-state methods; strategies improving BPTI sampling may not capture apo–holo dynamics.Source: §3.2.1 · p. 13, §3.2.2 · p. 14

3.2.3 DISTRIBUTION PREDICTION

• Conformation-sampling generative models outperform perturbation-based methods, but fine-tuning on MD data trades sample quality against diversity and performance, and a gap to long-term MD sampling remains.AlphaFlow and ConfDiff fine-tuned on MD data reach performance comparable to 2.5 ns MD simulations on some metrics.Source: §3.2.3 · pp. 14–15

4 CONCLUSIONS AND FUTURE WORK

• The study evaluates protein foundation models across eight tasks, particularly protein design and conformation dynamics, using a unified multi-metric framework.The authors present the framework as a basis for insights into model development and use.Source: §4 · p. 15

4.1 KEY OBSERVATIONS

• ProteinBench combines reconstitution and physical-rationality metrics and measures metrics on training data to provide a high-resolution reference for generative targets.The authors caution that some tasks may still lack sufficiently accurate evaluation methods because of limitations in complex structure prediction.Source: §4.1 · p. 15
• Language and structure-based models show different strengths across protein design objectives, and no single model excels across all objectives.Language models perform well in native sequence recovery and scaffolding evolution-conserved motifs, while structure-based models show greater robustness to structural noise in de novo design and potential for new folds or functions.Source: §4.1 · p. 16
• Conformation prediction remains challenging for multiple-state and distribution prediction, and fine-tuning on MD conformations showed little to no improvement on apo-holo prediction.The authors also note a common trade-off between diversity and quality in conformation prediction.Source: §4.1 · p. 16

4.2 LIMITATIONS AND FUTURE WORK

• The benchmark's model selection may be incomplete, training-data inconsistencies hinder direct architecture comparisons, and its task coverage could be expanded.The authors describe continued refinement and expansion as future work.Source: §4.2 · p. 16

A OVERVIEW OF PROTEIN FOUNDATION MODEL BENCHMARKS

• Most existing benchmarks focus narrowly on task-specific evaluations rather than holistic assessment of protein foundation models.The benchmark landscape is described as limited in scope and applicability.Source: A OVERVIEW OF PROTEIN FOUNDATION MODEL… · p. 16
• Existing benchmarks often overlook backbone design, sequence design, and structure-sequence co-design, while few address single-state and multiple-state prediction.Source: A OVERVIEW OF PROTEIN FOUNDATION MODEL… · p. 17
• ProteinBench covers inverse folding, backbone and sequence design, structure-sequence co-design, antibody design, single-state folding, and multiple-state prediction.The authors describe it as having the most comprehensive task coverage among the benchmarks considered.Source: A OVERVIEW OF PROTEIN FOUNDATION MODEL… · p. 17

B.1 ADDITIONAL RESULTS FOR PROTEIN DESIGN

• Backbone-design results for 200- and 400-amino-acid sequences show trends similar to those at other lengths.The results are presented in Table 12.Source: §B.1 · p. 17

B.2 ANTIBODY DESIGN

• The antibody-design appendix introduces its evaluation concept, dyMEAN variant, training and testing datasets, and metric implementation.Source: §B.2 · p. 17

B.2.1 EVALUATION CONCEPT

• Antibody design is evaluated through generated CDR-H3 sequences and predicted structures, focusing on relative model performance rather than in vivo or in vitro functionality.For methods generating multiple antibodies per antigen, the evaluation uses 64 sequences and reports average performance and standard deviation across samples.Source: §B.2.1 · p. 17

B.2.2 VARIANT OF DYMEAN

• dyMEAN-FixFR receives the true non-CDR-H3 structure and binding pose, uses Rosetta side-chain repacking, and randomizes initialization to generate multiple antibodies per antigen.These modifications align dyMEAN more closely with other methods for comparison.Source: §B.2.2 · pp. 17–18

B.2.3 DATASET

• SAbDab training data were deduplicated by CDR-L3 and CDR-H3, clustered at 40% CDR-H3 sequence similarity, and clusters containing RAbD complexes were excluded before a 9:1 split.The resulting training and validation sets contain 1,786 and 193 complexes, respectively.Source: §B.2.3 · p. 18
• The training complexes were restricted to heavy- and light-chain antibodies paired with protein antigens before deduplication and clustering.Source: §B.2.3 · p. 18
• The test set contains 55 antibody-antigen complexes selected from RAbD, requiring both antibody chains and at least one protein antigen chain.The original 60 complexes were reduced by chain/antigen criteria, incorrect chain information, and an HERN execution error.Source: §B.2.3 · p. 18

B.2.4 IMPLEMENTATION OF EVALUATION METRICS

• AAR counts generated CDR-H3 residues matching the natural antibody, while RMSD and TM-score compare generated and natural CDR-H3 structures.RMSD uses CA coordinates; for dyMEAN, FR residues at both CDR-H3 ends are aligned before calculating it, and TM-score is calculated only for CDR-H3.Source: §B.2.4 · p. 18
• Binding Energy is calculated with Rosetta InterfaceAnalyzer after side-chain packing and CDR-H3 minimization against the antigen while holding the backbone fixed.The reported minimization used 100 steps.Source: §B.2.4 · p. 19
• SeqSim averages matched-residue similarity across generated sequence pairs, measuring between-antigen SeqSim-outer and, for multi-sample methods, within-antigen SeqSim-inner.Alignment scores matches as 1 and gaps or mismatches as 0, with terminal gaps disallowed because CDR-H3 ends are fixed to FR3 and FR4.Source: §B.2.4 · p. 19
• PHR measures hydrophobic-residue proportion, while CN-Score averages KDE densities fitted to natural CDR-H3 peptide-bond lengths.The narrow natural bond-length distribution makes CN-Score fall sharply for slight deviations from its 1.3310 average.Source: §B.2.4 · pp. 19–20
• Clashes-inner counts short CA-CA distances within generated CDR-H3, whereas Clashes-outer counts such pairs between CDR-H3 and surrounding components.The threshold is 3.6574, derived from CA-CA distance statistics in the RAbD CDR-H3 dataset.Source: §B.2.4 · p. 20
• SeqNat uses AntiBERTy pLL for the CDR-H3 region while supplying the entire heavy-chain sequence as model input.Source: §B.2.4 · p. 20
• Total Energy sums Rosetta full-atom residue scores over CDR-H3 after the designed region undergoes the same energy optimization used in the Functionality section.Source: §B.2.4 · p. 20

B.3.1 DATASETS •

• The conformation-prediction datasets include CAMEO2022, Apo-holo, BPTI, and an 82-protein post-May 2019 subset of ATLAS.CAMEO2022 contains 183 chains; Apo-holo contains 91 proteins, and BPTI is 58 amino acids long.Source: §B.3.1 · pp. 20–21

B.3.2 EVALUATION METRICS

• Structural accuracy compares generated conformations with reference structures using TM-score, RMSD, GDT-TS, and lDDT.Source: §B.3.2 · p. 21
• For multiple-state prediction, each state is scored by its best generated-sample accuracy, while ensemble accuracy averages accuracy across states.The passage gives apo-TM as the highest TM-score among 20 samples against the apo reference.Source: §B.3.2 · p. 21
• Distribution prediction evaluates flexibility by correlation, distributional accuracy by Wasserstein distances, and ensemble observables by Jaccard similarity and mutual information.The compared observables include residue-residue contacts and residue exposures.Source: §B.3.2 · p. 21
• Diversity is average pairwise structural similarity among generated samples, while structural quality violations include CA clashes, CA breaks, and peptide-bond breaks.CA-breaks use a connecting-residue distance threshold above 4.2 Å; peptide-bond breaks use a C-N threshold of 1.4 Å.Source: §B.3.2 · pp. 21–22

B.3.3 MODEL IMPLEMENTATIONS

• Folding-model baselines use model-specific sampling and selection, with AlphaFold2, OpenFold, and RoseTTAFold2 run without templates and ESMFold producing one deterministic structure per protein.EigenFold samples five structures and selects by ELBO; AlphaFold2 and OpenFold select by pLDDT.Source: §B.3.3 · p. 22

B.3.4 EXTENDED DISCUSSION ON PROTEIN CONFORMATION MODELS

• MSA-subsampling perturbs folding-model input by reducing MSA depth, allowing alternative folded structures without retraining the model.The benchmark implements this approach with OpenFold and varies MSA-depth parameters.Source: §B.3.4 · p. 23
• Str2Str instead perturbs a predicted structure with a structure-only diffusion model, controlling perturbation through maximum diffusion time and sampling at times up to that limit.Source: §B.3.4 · p. 23
• EigenFold, AlphaFlow, and ConfDiff fine-tune generative models using PDB structural data, while AlphaFlow and ConfDiff also fine-tune on densely sampled MD conformations.The passage notes that models trained solely on PDB data are limited in predicting conformational distributions.Source: §B.3.4 · p. 23
• ConfDiff adds classifier-free guidance or energy/force guidance, but its physical prediction modules are dataset-specific and require additional training.Source: §B.3.4 · p. 23

• The section reports complete results for multiple-state prediction on BPTI and apo-holo data, plus distribution prediction on ATLAS.Source: §B.3.5 · p. 23
• BPTI results cover RMSD-based accuracy at sample sizes from 10 to 1,000, with diversity and quality evaluated across 1,000 conformations.Source: Table 13 · p. 24
• Apo-holo results compare sampled conformations with reference structures using maximum TM-score, summarized across 91 proteins.Source: Table 14 · p. 24
• ATLAS distribution-prediction results report median metrics across 82 proteins, with some metrics unavailable for models lacking sufficient resolution.Source: Table 15 · p. 25