PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking Multimodal Protein Language Models at Inference Time

Yi Zhou, Qipeng Wang, Yunqing Liu, Jun Xia, Qing Li, Wenqi Fan

arXiv:2608.25855v1cs.CEcs.AI

TL;DR

The paper addresses limited empirical understanding of inference-time strategies for multimodal protein language models. It systematically evaluates distribution-level sampling, logit-level guidance, and trajectory-level search across three models and four tasks, finding consistent gains over default configurations and task-oriented inference preferences.

  • Problem

    Prior work has focused more on training than on how inference-time strategies behave, while default protocols are coarse and misaligned across multimodal pLMs, hindering fair comparison.

  • Method

    The paper evaluates vanilla sampling, task-specific classifier-free guidance, and reward-guided search across three multimodal pLMs and four protein modeling tasks, controlling distributions, per-step logits, and parallel trajectories.

  • Results

    Inference strategies consistently improve empirical performance over default configurations across tasks, bringing multimodal pLMs close to or beyond task-specific specialist models.

  • Takeaways & Limitations

    Inference protocol is a hidden confounder in protein-modeling evaluation, and inference preferences are largely task-oriented.

  • Takeaways & Limitations

    The results concern controlled benchmark tasks and do not directly establish broader biological utility or experimental success.

Abstract

from arXiv · show

Multimodal protein language models (pLMs) learn joint protein sequence-structure distributions, and their generation performance should also depend critically on inference-time sampling strategies. Yet prior work has focused more on model training than on how inference-time strategies behave. In this paper, we establish a three-stage investigation framework to empirically study the inference design space of multimodal pLMs across three representative pLMs and four fundamental tasks. We evaluate vanilla sampling, task-specific classifier-free guidance, and reward-guided beam search on multimodal pLMs, corresponding to controls over sampling distributions, per-step logits, and parallel trajectories. Throughout the complementary advancements centered on exploration-exploitation trade-off, we (1) reveal the suboptimality of default inference protocols and identify task-oriented sampling preferences; (2) observe substantial quantitative gains across tasks, consistently boosting the upper bound performance of multimodal pLMs without updating model parameters; (3) derive conclusions about base models that differ from prior consensus.

1 Introduction

Multimodal pLMs model protein sequence and structure jointly, but inference-time sampling can substantially alter their outputs. The paper investigates this underexplored design space and shows that standardized strategies improve performance and change conclusions about model capabilities.

  • Motivation: Inference-time sampling can produce sharply different structure predictions from the same conditional point, depending on trajectory length and exploration.ESM3’s one-step argmax prediction generally preserves the fold, whereas iterative temperature-based sampling can collapse.
  • Research gap: Prior multimodal pLM research emphasizes training and fine-tuning while largely leaving inference-time sampling strategies underexplored.Existing models use model- and task-specific vanilla defaults without widely recognized guidance or reward-guided search techniques.
  • Approach: The investigation controls sampling at three levels: distributions, per-step logits, and parallel trajectories.It benchmarks vanilla sampling, introduces task-specific classifier-free guidance, and studies trajectory-level reward-guided search.
  • Findings: Across four tasks, the three-level inference strategies consistently advance the best empirical performance of multimodal pLMs.The strongest improvements include ESM3’s unconditional sequence-structure cogeneration and motif-scaffolding.
  • Findings: Standardized inference strategies can make models viewed as weak under default sampling competitive, revealing inference protocol as a hidden evaluation confounder.This finding directly affects cross-model comparisons that rely on unaligned defaults.

2 Preliminaries

Multimodal pLMs represent protein sequence and structure as parallel token tracks and generate them through iterative diffusion sampling. Inference combines sampling schedules, masking operations, temperature control, classifier-free guidance, and trajectory search.

  • Modeling and benchmarks: Multimodal pLMs model proteins with parallel sequence and structure token tracks and support diverse tasks through iterative sampling from pLM logits.The benchmark covers structure prediction, inverse folding, motif scaffolding, and unconditional sequence-structure cogeneration.
  • Diffusion sampling: The two diffusion timelines support sequence-to-structure and synchronous sampling schedules.Sequence-to-structure completes one timeline before advancing the other, whereas synchronous sampling couples them under a shared timeline.
  • Diffusion sampling: Each diffusion step selects masked positions for unmasking, optionally remasks low-confidence tokens, and samples tokens from temperature-scaled categorical distributions.Unmasking may be deterministic, stochastic, or random; remasking enables later refinement, while temperature controls randomness versus greediness.
  • Guidance and search: Classifier-free guidance steers diffusion sampling toward context-aligned high-likelihood regions by modifying per-step logits.Guidance scale w > 1 amplifies the conditional signal and sharpens adherence to specified constraints.
  • Guidance and search: Reward-guided search explores multiple trajectories in parallel and evaluates them using global rewards, combining exploration with trajectory selection.The framework unifies parallel exploration and advance evaluation through generalized beam search.

3 Vanilla Sampling

Vanilla sampling defaults are not consistently aligned or empirically justified across multimodal pLMs. Standardized grid search reveals task-specific exploration-exploitation preferences and improves benchmark performance without changing model parameters.

  • Motivation: Default inference implementations differ across models and rely on coarse heuristics whose rationality and optimality are unverified.These discrepancies hinder fair cross-model comparison and motivate a unified sampling framework.
  • Method: The benchmark searches six sampling dimensions: diffusion steps, unmasking, remasking, temperature, temperature annealing, and multimodal sampling order.The grid spans multiple denoising lengths, three unmasking modes, remasking choices, and temperatures from 0 to 1.
  • Results: Enabling synchronous sampling, stochastic unmasking, and remasking makes ESM3 considerably more competitive on unconditional sequence-structure cogeneration.These settings improve the sampling trajectory relative to default sequence-to-structure sampling.
  • Exploration-exploitation trade-off: Structure prediction and inverse folding favor short, low-temperature trajectories, whereas unconditional cogeneration favors long, stochastic, high-temperature trajectories.Motif scaffolding occupies a balanced regime with long trajectories, moderate temperature, and synchronous decoding.

4 Reward-Free Guidance

The paper evaluates classifier-free guidance (CFG) as a per-step logit control for multimodal protein tasks, building on optimized vanilla sampling. CFG improves ESM3 substantially, while gains depend on task solution-space breadth and how strongly base models encode conditional signals.

  • CFG steers per-step logits toward condition-aligned regions, complementing vanilla sampling’s control over the foundational sampling distribution.The framework adapts guidance to multimodal tracks and task-specific conditions.
  • Unconditional Protein Sequence-Structure CoGeneration: Cross-modal CFG improves designability and diversity across all three base models in unconditional sequence-structure co-generation.For ESM3, average pLDDT reaches 88.748, scTM reaches 0.931, and designable samples rise to 325.0 from 174.8 under optimal vanilla sampling.
  • Motif-Scaffolding: ESM3 with CFG solves 23 of 24 motif-scaffolding cases in 4 of 5 runs, reaching a 37.5% average success rate and 179.4 unique solution clusters.These values nearly double default success rate and diversity, respectively.
  • Protein Structure Prediction: ESM3 benefits clearly from CFG in structure prediction, reaching RMSD 5.144 and TM-score 0.870 on CAMEO2022, and RMSD 3.969 with TM-score 0.889 on PDB Date.DPLM-2 and DPLM-2.1 show modest RMSD improvements but no consistent TM-score gains.
  • Inverse Folding: CFG improves structural consistency in inverse folding across base models, while ESM3 reaches average scTM 0.910 on CAMEO2022 and 0.954 on PDB Date.AAR declines slightly, consistent with inverse folding’s one-to-many mapping rather than a requirement to recover the native reference.
  • CFG helps most when conditions leave a larger solution space and when the base model has strongly internalized the corresponding conditional signal.The paper links larger gains to exploratory tasks such as unconditional co-generation and motif-scaffolding, and stronger steering capacity to stronger multimodal models.

5 Reward-Guided Search

Reward-guided search extends inference control from local token logits to parallel trajectories scored by global, model-internal rewards. On ESM3, this stage improves results across four tasks, while incurring considerable inference overhead.

  • Search procedure: Reward-guided search maintains parallel trajectories, expands each into candidates, scores partially masked candidates with estimated rewards, and prunes back to a fixed beam width.Expansion and selection occur every K steps; after the final step, one output is selected from the retained beam.
  • Reward design: Model-internal pTM and foldability pTM scores provide rewards for structure and sequence tracks while preserving the multimodal model’s all-in-one setting.Structure tokens receive structural pTM directly, whereas sequences are greedily folded into structure tokens before scoring.
  • Results: Across all four tasks, beam search improves over classifier-free guidance and enables ESM3 to match or surpass task-specific state-of-the-art systems without task-specific training or external rewards.The reported gains build on tuned vanilla sampling and guidance choices.
  • Results: 23 of 24 motif-scaffolding problems are solved in every run, while success rate rises from 37.5% to 53.3% and unique clusters from 179.4 to 245.2.For unconditional co-generation, threshold-based random selection increases unique designable clusters from 116.6 to 139.0.
  • Efficiency: Beam search incurs considerable computational overhead at inference to achieve improved benchmark performance.The search stage focuses on performance rather than computational efficiency.

6 Conclusion

The paper finds that inference strategies act complementarily across three base models and four tasks, moving multimodal pLMs close to or beyond specialist performance. It frames this progress as a bottom-up navigation of exploration and exploitation.

  • 6 Conclusion: Across three base models and four tasks, vanilla sampling, reward-free guidance, and reward-guided search produce consistent gains over default configurations.The strategies control the sampling distribution, logits, and trajectories at progressively broader levels.
  • 6 Conclusion: Inference preferences are largely task-oriented, and exploration-exploitation can be navigated from sampling distributions to logits to global trajectory selection.This sequence summarizes the paper’s three-stage inference framework.

Limitations

The study evaluates fixed multimodal pLMs on controlled benchmarks, so its findings do not directly establish broader biological utility or experimental success. Efficiency analysis is also limited because optimization targets performance rather than computational cost.

  • Scope: Results on controlled benchmark tasks do not directly translate to broader biological utility or experimental success.The study does not examine interactions with external models, human experts, laboratory feedback, or multi-stage design pipelines.
  • Efficiency: Performance-efficiency analysis remains basic because inference protocols are optimized for benchmark performance rather than computational efficiency.Reported FLOPs and runtime reflect performance-optimized protocols; multiobjective optimization under limited resources remains future work.

Ethical Considerations

The work supports scientific, clinical, and biomedical research, while generative biology creates a theoretical misuse risk. The authors characterize that risk as mitigated by the study’s computational scope and lack of hazardous-design validation.

  • Ethical Considerations: Generative biology carries a theoretical risk of misuse for designing harmful biomolecules, including pathogenic proteins.The authors identify this as the principal societal risk of the work.
  • Ethical Considerations: The risk is described as mitigated because the study investigates inference procedures on established benchmarks without autonomously proposing or experimentally validating hazardous designs.All data used are publicly available, and no ethical approval was required.

A Appendix

The appendix provides supplementary details on model inputs, evaluation, sampling strategies, guidance, reward-guided search, and inference efficiency.

  • Supplementary preliminaries describe protein structure tokenization and base model implementations.
  • Additional sections document evaluation pipelines, vanilla sampling, reward-free guidance, and reward-guided search.
  • The appendix also includes hyperparameter analyses, ablations, detailed structure prediction, and inference efficiency analysis.

A.1.1 Protein Structure Tokenization

Multimodal pLMs represent protein sequences as categorical amino-acid variables and convert continuous protein structures into discrete tokens for language-model processing.

  • A protein sequence is represented as s = (s1, s2, . . . , sL), with each si denoting the identity of one residue.The residue identities generally use the 20 standard amino acids.
  • Protein structure begins as per-residue atom coordinates x = (x1, x2, . . . , xL), where xi ∈Rni×3.
  • Quantization-based tokenizers convert structure coordinates into discrete tokens compatible with language models.The process is framed under a VQ-VAE-style encoder, quantizer, and decoder pipeline.
  • The decoder reconstructs 3D coordinates and can compute structural quality scores such as predicted TM-score.

A.1.2 Base Model Descriptions

The appendix describes the evaluated multimodal pLMs, benchmark tasks, and inference analyses, including model-specific sampling behavior, guidance effects, and search trade-offs.

  • Base Model Descriptions: DPLM-2 extends a sequence-only pLM to jointly model protein sequence and structure, while DPLM-2.1 predicts binary token bits with the same tokenizer.Both use pretrained 650M checkpoints in the implementation.
  • Evaluation Tasks: The study evaluates unconditional sequence-structure cogeneration, motif scaffolding, structure prediction, and inverse folding with task-specific benchmark criteria.Motif scaffolding requires both global designability and local motif preservation.
  • Vanilla Sampling: Default inference configurations imply longer, more random sampling for unconditional cogeneration and motif scaffolding than for structure prediction or inverse folding.
  • Reward-free Guidance: 473.0 aligned PDB samples and mean PDB-TM 0.821 ± 0.156 are obtained with ESM3 cross-modal CFG, versus 387.2 and 0.548 ± 0.316 under optimal vanilla sampling.
  • Reward-free Guidance: Full dual-conditioning CFG outperforms either single-conditioning variant, showing that explicit motif and implicit cross-modal guidance are complementary.
  • Reward-guided Search: Reward-guided search maintains parallel trajectories, expands them independently, and selects candidates using task-aligned global rewards.
Loading 2608.25855v1…