Protein Autoregressive Modeling via Multiscale Structure Generation

Yanru Qu; Cheng-Yen Hsieh; Zaixiang Zheng; Ge Liu; Quanquan Gu

Source-linked AI summary

Protein Autoregressive Modeling via Multiscale Structure Generation

Yanru Qu, Cheng-Yen Hsieh, Zaixiang Zheng, Ge Liu, Quanquan Gu

arXiv:2602.04883v2Feb 4, 2026cs.LGcs.AIq-bio.BMq-bio.QM

TL;DR

Protein backbone autoregression must handle continuous coordinates, bidirectional residue dependencies, and exposure bias. PAR uses multiscale next-scale prediction with transformer conditioning and flow-based decoding, achieving strong unconditional generation, zero-shot conditional tasks, and favorable sampling behavior.

Problem
Protein autoregressive modeling is limited by continuous structural data, bidirectional residue dependencies, and training-inference mismatch from exposure bias.
Method
PAR uses hierarchical multi-scale downsampling, an autoregressive transformer for scale-wise conditioning, and a flow-based decoder for direct Cα coordinate generation, with noisy context learning and scheduled sampling.
Results
PAR captures protein distributions with high design quality, achieves FPSD 161.0 against PDB, supports zero-shot prompt generation and motif scaffolding, and reaches 2.5x sampling speedup over single-scale baselines.
Takeaways & Limitations
PAR provides a general multi-scale framework for protein backbone generation that combines coarse-to-fine structural modeling with zero-shot conditional generation.
Takeaways & Limitations
The work focuses on backbone Cα atoms and leaves conformational dynamics modeling and extension to full-atom representations for future research.

Abstract

from arXiv · show

We present protein autoregressive modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation via coarse-to-fine next-scale prediction. Using the hierarchical nature of proteins, PAR generates structures that mimic sculpting a statue, forming a coarse topology and refining structural details over scales. To achieve this, PAR consists of three key components: (i) multi-scale downsampling operations that represent protein structures across multiple scales during training; (ii) an autoregressive transformer that encodes multi-scale information and produces conditional embeddings to guide structure generation; (iii) a flow-based backbone decoder that generates backbone atoms conditioned on these embeddings. Moreover, autoregressive models suffer from exposure bias, caused by the training and the generation procedure mismatch, and substantially degrades structure generation quality. We effectively alleviate this issue by adopting noisy context learning and scheduled sampling, enabling robust backbone generation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible human-prompted conditional generation and motif scaffolding without requiring fine-tuning. On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality, and exhibits favorable scaling behavior. Together, these properties establish PAR as a promising framework for protein structure generation.

1 Introduction

Protein autoregressive modeling has been limited by continuous structural data and bidirectional residue dependencies. PAR addresses these challenges with coarse-to-fine next-scale generation, exposure-bias mitigation, and flexible generation capabilities.

Motivation: Protein backbone autoregression is challenged by discretization of continuous atomic positions and the bidirectional spatial dependencies between residues.These limitations can reduce structural fidelity and constrain the quality of autoregressive structure generation.
PAR framework: PAR predicts protein structures across multiple scales, conditioning each finer scale on previously generated coarser scales.This design follows the hierarchical organization of proteins, from global topology to local and atomic details.
PAR framework: The model combines multi-scale downsampling, an autoregressive transformer, and a shared flow-based decoder that directly models Cα backbone atoms.The transformer produces scale-wise conditional embeddings, while the decoder denoises each scale conditioned on those embeddings.
Training: Noisy context learning and scheduled sampling mitigate exposure bias caused by training on ground-truth context while inference uses model predictions.Exposure bias can cause errors to accumulate across autoregressive steps and degrade generated structure quality.
Results: PAR forms global topology at coarse scales and progressively refines local details, producing an interpretable sculpting-like generation process.The approach supports zero-shot prompt-based generation and motif scaffolding, while unconditional generation shows favorable scaling and competitive FPSD results.
Results: PAR achieves an FPSD score of 161.0 against the PDB dataset and a 2.5x sampling speedup compared with single-scale baselines.The reported results also indicate that performance scales with training compute.

2 Background and Related Work

Protein structure generation has mainly used flow or diffusion models, while autoregressive modeling has been less established for continuous structural data. Related autoregressive approaches face discretization and ordering challenges when adapted beyond sequences.

Flow and diffusion-based structure generative models: Flow- and diffusion-based protein generators transform prior samples toward target distributions using frame-based representations or direct atom-coordinate modeling.These approaches have been widely applied to protein backbone generation.
Autoregressive modeling: Autoregressive modeling predicts each next token from preceding tokens and has shown scalability and zero-shot generalization in language and vision.Its standard formulation relies on a unidirectional sequence.
Autoregressive modeling: Adapting autoregressive models to continuous generation often uses tokenizers that discretize data and may discard fine-grained details.Continuous-valued conditioning for image generation offers one precedent for avoiding direct discretization.

3 Protein Autoregressive Modeling

PAR models protein backbones autoregressively across hierarchical scales, generating coarse structure first and refining it to full-resolution atomic coordinates. Its multi-scale transformer–flow architecture avoids discretization and residue-wise ordering while training strategies mitigate exposure bias.

Multi-scale structure generation: PAR decomposes each protein backbone into n coarse-to-fine scales and learns pθ(xi | X<i) through next-scale autoregression.The final scale xn equals the full backbone x, while each scale is conditioned on preceding coarser representations.
Multi-scale structure generation: Hierarchical downsampling interpolates protein coordinates along the sequence dimension to create coarse structural layouts used as training context and targets.The deterministic decomposition avoids marginalization, and the resulting factorization is pθ(x) = ∏i pθ(xi | X<i).
Coarse-to-fine backbone modeling: Next-scale prediction captures mutual structural dependencies across scales instead of imposing residue-wise unidirectional ordering.The approach is motivated by spatially close residues that can interact despite being distant in sequence.
Coarse-to-fine backbone modeling: The autoregressive transformer encodes preceding scales into conditional embeddings, while a flow-based decoder directly models Cα coordinates in continuous space.This design avoids discretizing protein structures into tokens and conditions each scale’s atomic generation on its learned embedding.
Coarse-to-fine backbone modeling: At inference, PAR repeatedly upsamples the current structure, predicts the next-scale embedding, and samples the next scale until generating the full-resolution backbone.ODE or SDE sampling is used by the flow-matching decoder, with KV caching applied throughout the autoregressive process.
Mitigating exposure bias: Noisy context learning and scheduled sampling expose training to imperfect prior-scale predictions, addressing exposure bias from teacher-forced training and inference-time self-conditioning.Exposure bias creates a training–inference mismatch in which errors can accumulate across autoregressive steps and degrade output quality.

4 Experiments

Experiments evaluate PAR’s unconditional generation, interpretable coarse-to-fine behavior, zero-shot conditioning, scaling, sampling efficiency, exposure-bias mitigation, and architectural choices. PAR supports prompt-based layout control and motif scaffolding, while larger decoders, multiscale sampling, and training strategies improve generation quality or efficiency.

Experimental scope: PAR is evaluated on unconditional backbone generation, zero-shot generalization, scaling behavior, efficient sampling, exposure-bias mitigation, and additional ablations.The evaluation compares PAR with existing generative methods and studies its multiscale design empirically.
Generation process: PAR generates structures coarse-to-fine, establishing global layout at coarser scales before adding local details at finer scales.This produces an interpretable generation process analogous to sculpting a statue.
Zero-shot task generalization: Small point prompts guide global backbone arrangements without fine-tuning, while motif-coordinate teacher forcing enables zero-shot scaffolding without additional conditioning or fine-tuning.Longer prompts can provide finer-grained control, and generated scaffolds preserve supplied motifs.
Scaling behavior: 187 FPSD against PDB and 170 against AFDB are reported, while fS increases with model size and computational budget and larger models lower sc-RMSD.A 60M autoregressive transformer is sufficient for strong evaluation results, allowing capacity to be prioritized for the backbone decoder.
Exposure-bias mitigation: Noisy context learning improves sc-RMSD from 2.20 to 1.58, and adding scheduled sampling further improves the reported designability metric to 1.48.These strategies align training more closely with inference and are evaluated with a 60M PAR model trained for 100K steps.
Multiscale ablations: Three scales outperform two, whereas four or five scales degrade designability, potentially because of error accumulation and exposure bias.The by-length downsampling strategy also outperforms the by-ratio strategy under comparable upsampling ratios.
Architectural ablations: Autoregressive transformer encoding produces more structurally consistent guidance across scales than directly feeding prior-scale structures into the decoder.The comparison uses downsampled test structures as prompts and evaluates re-upsampling consistency.

5 Discussion

PAR is presented as a first multi-scale autoregressive framework for protein backbone generation, addressing standard autoregressive limitations while supporting conditional and unconditional generation. Its discussion highlights zero-shot control, structural fidelity, and future extensions beyond backbone generation.

PAR addresses unidirectional dependency, discretization, and exposure bias through multi-scale structural modeling.
PAR supports zero-shot conditional generation, including coarse point-based structural layouts and atomic-coordinate motif scaffolding.
PAR exhibits distributional fidelity and high generation quality for unconditional backbone generation.
Conformational dynamics modeling and full-atom generation remain open directions beyond the current backbone-focused work.The paper states that PAR focuses on backbone Cα atoms and leaves conformational-distribution modeling for future research.

A Implementation and Evaluation Details

PAR training follows the Proteina implementation and compares flow-based decoders at multiple parameter scales under matched training settings. The autoregressive module uses a smaller configuration, while model specifications are summarized in the accompanying hyperparameter table.

PAR training follows Proteina’s architecture and hyperparameter setup.
Training uses 8 H100 GPUs, batch size 15 per GPU, and 200k optimization steps.
Flow-based backbone decoders are trained with 60 M, 200 M, and 400 M parameters.
The autoregressive module uses Proteina’s 60 M-parameter configuration, while Proteina is retrained from scratch for fair comparison.
Table 6 lists PAR hyperparameters for the autoregressive transformer Tθ and flow-based atomic decoder vθ.

A.1 Implementation Details

PAR uses scale-specific and interpolated positional representations to share a decoder across structural resolutions. These embeddings preserve information about scale and relative residue location across coarse-grained inputs.

Scale embedding assigns each structural resolution a unique scale ID to distinguish its statistical characteristics.
Interpolated position embeddings derive coarse coordinates and relative position IDs from neighboring real residues in the original structure.
Across different input lengths and scales, interpolated positions preserve each residue’s relative location within the original protein.

A.2 Evaluation Metrics

The evaluation measures generation quality, diversity, novelty, designability, secondary structure, and distributional fidelity across sampled protein backbones. Protocols combine established structural metrics with sequence-design and folding-based assessments.

The evaluation samples 100 structures at each of five sequence lengths—50, 100, 150, 200, and 250—for 500 structures total.
Designability generates 8 ProteinMPNN sequences per structure, folds them with ESMFold, and counts structures with minimum RMSD below 2 Å.
Diversity is the average pairwise TM-score among designable structures, averaged across the five sequence lengths.
Secondary-structure evaluation reports average alpha-helix and beta-sheet proportions among designable structures.
For distributional evaluation, 5,000 structures are sampled across lengths 60–255 in steps of 5 without designability filtering.
FPSD measures Wasserstein distance between generated and reference structure distributions in fold-predictor feature space.
Protein Fold Score measures sample-level fold prediction confidence together with diversity across predicted fold classes.
Novelty uses maximum TM-score against Foldseek’s precomputed PDB database for each generated structure.

B Datasets

The evaluation spans sampling, self-conditioning, long-protein generation, diversity, and runtime, with analyses of how sampling choices affect designability. PAR performs better at shorter long-protein lengths, while designability degrades for longer chains.

Sampling: Combining SDE sampling at the first scale with ODE sampling at later scales reduces sampling steps without harming designability.Aggressively reducing SDE steps or using ODE across all scales produces much worse designability.
Self-conditioning: Self-conditioning consistently improves sc-RMSD across protein-length ranges and is complementary to multi-scale generation.The ablation uses the same 60M PAR model across varying lengths.
Long proteins: PAR shows higher designability at lengths 300 and 400, while both PAR and Proteina degrade at lengths 500–700.The authors associate the longer-length degradation with fewer training samples in the 500–700 residue range.
Diversity: Increasing sampling stochasticity raises foldseek cluster diversity, peaking at γ=0.6.The analysis generates 500 structures across lengths 50, 100, 150, 200, and 250 using a TM-score threshold of 0.5.

C.5 Zero-shot Motif Scaffold Benchmark

PAR is evaluated for zero-shot motif scaffolding using fixed motif coordinates and generated surrounding structures. Unlike the cited baselines, PAR requires no fine-tuning or additional motif conditioning, while flexible scale inference can reduce designability.

Evaluation protocol: PAR generates 1000 backbone structures per benchmark problem and evaluates motif scaffolds using ProteinMPNN and ESMFold.Success requires scRMSD ≤2Å, motifRMSD ≤1Å, pLDDT ≥70, and pAE ≤5.
Benchmark setting: PAR is the only method evaluated zero-shot, whereas the other baselines use training or finetuning with additional motif conditioning.The benchmark table denotes the zero-shot model as PAR* and reports success rate.
Zero-shot scaffolding: Without fine-tuning or conditioning, PAR generates plausible scaffolds that preserve finer-grained motif prompts such as atomic coordinates.Ground-truth motif coordinates are teacher-forced at each scale before propagation to the next scale.
Scale-agnostic inference: Using an unseen scale configuration keeps FPSD stable but substantially reduces designability and structural detail.A 3-scale model is evaluated with five scales after discarding the fixed scale embedding.

C.7 Ablating AR and Decoder Size

Ablations examine decoder capacity, autoregressive context encoding, exposure bias, spatial preservation, and attention patterns. The results emphasize decoder capacity and multi-scale context handling for reliable coarse-to-fine generation.

Decoder design: A 200M autoregressive module with a 20M per-token diffusion head produces average sc-RMSD of 16 and fails to generate reasonable structures.The authors attribute this likely failure to insufficient global correlation modeling for the coarse first scale.
Model capacity: Using a large decoder effectively improves generation quality, whereas increasing the AR transformer from 60M to 400M does not.The decoder-size comparison is reported in Table 13 and the scaling experiments.
Exposure bias: Larger AR models may intensify exposure bias because they overfit more strongly to ground-truth context during training.The paper identifies limited data and protein modeling’s sensitivity to small errors as additional concerns.
Spatial preservation: Sequence-based downsampling slightly increases pairwise-distance RMSE at coarser scales, while LDDT remains consistently 1 and RMSE stays low.The results indicate preservation of essential pairwise spatial correlations despite small information loss.

C.10 Training Time Comparison

The section compares training-time reporting and discusses extensions of PAR beyond backbone generation. Proposed directions include all-atom design, molecular ensembles, multi-chain structures, and other biomolecules.

Training-time reporting: Training time is reported for 100k steps on 8 H100 GPUs with batch size 15 and diffusion multiplicity 2.The provided passages identify the reporting setup but not a measured duration.
Future directions: An all-atom extension could add backbone-atom and side-chain scales, with residue-type prediction as an additional output head.Backbone atoms would be conditioned on Cα and coarser scales, followed by full side-chain generation.
Future directions: PAR might be extended to protein ensembles by training on ensemble data with sequence conditioning and coarse-to-fine one-to-many generation.The proposed formulation targets molecular-dynamics-related ensemble generation.
Future directions: Multi-chain generation could introduce chain identifiers, while other biomolecules require suitable coarse-grained representations.The authors connect the multi-scale formulation to interaction modeling and binder design.