PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative modeling of intrinsically disordered protein regions by reinforcing sparse autoencoder features

Jason X. Liu, Sebastian Ibarraran, Frank Hu, Soojung Yang, Xinyu A. Feng, Abigail Park, Anagha Aneesh, Lacramioara Bintu, Alexander R. Dunn, Grant M. Rotskoff

arXiv:2610.02189v1cs.LG

TL;DR

IDR design is challenging because structure-based approaches do not readily apply and full-length protein language models are biased toward folded domains. The paper trains IDiom on predicted IDRs and introduces RL-SAE to target interpretable sequence features; across design tasks it improves feature activation and predicted biological properties relative to steering and supervised fine-tuning.

  • Problem

    Structure-based methods do not readily apply to IDRs, and full-length protein language models have generative priors biased toward folded domains; explicitly controlling interpretable sequence patterns is also challenging.

    Source: §1 · pp. 1–2

  • Method

    IDiom is trained on 54M predicted IDRs, and RL-SAE post-trains the model to reward activation of specified sets of SAE features.

    Source: §1 · p. 2

  • Results

    RL-SAE improves predicted localization and transcriptional activity over steering and supervised fine-tuning, activating 90% of targeted top-30 features on average versus 24% for activation steering.

    Source: §1 · p. 2, §5.2.2 · p. 6

  • Takeaways & Limitations

    RL-SAE supports selective incorporation and combination of function-associated sequence features within generated IDRs.

    Source: §1 · p. 2

    Evaluations are computational, so experimental validation is needed to confirm whether predicted properties translate into biological function.

    Source: §6 · p. 10

Abstract

from arXiv · show

Intrinsically disordered protein regions (IDRs) play central roles in cellular processes such as transcriptional regulation, signal transduction, and subcellular localization, yet their functional design remains challenging. Structure-based design methods do not readily apply to IDRs, and existing protein language models are trained on full-length protein sequences, thus learning a prior that is biased towards folded domains. Here, we present IDiom, an autoregressive protein language model trained on IDiom-DB, a dataset of 54 million predicted IDRs curated from the AlphaFold Database. IDiom generates diverse sequences that recapitulate the composition, patterning, motifs, and predicted disorder of natural IDRs. To control function-associated sequence patterns, we also introduce reinforcement learning with sparse autoencoder features (RL-SAE), a post-training method that rewards the generation of sequences that activate specified feature sets. Across eight IDR design tasks, RL-SAE sequences activate, on average, 90% of 30 targeted features, compared to 24% for activation steering. We demonstrate that RL-SAE improves the predicted subcellular localization and transcriptional activity of generated IDRs compared to steering and supervised fine-tuning, and enables features associated with distinct biological functions to be combined within individual sequences. Thus, IDiom and RL-SAE enable interpretable and composable IDR design through explicit control of function-associated sequence features. More broadly, RL-SAE could extend to other protein design settings where interpretable features provide useful design targets. Code is available at https://github.com/rotskoff-group/idiom.

1 INTRODUCTION

IDR design is difficult because structure-based approaches do not readily apply to disordered sequences, while full-length protein language models favor folded domains and do not explicitly control interpretable sequence patterns. The paper introduces IDiom and RL-SAE to address these gaps.

1 INTRODUCTION

  • IDRs support diverse cellular functions, but their rational design remains challenging because they do not adopt well-defined folds.The cited passage describes roles including flexible linkers, signaling hubs, and drivers of condensate formation.Source: §1 · pp. 1–2
  • IDiom is pretrained on 54M predicted IDRs and generates diverse sequences recapitulating natural IDR composition, patterning, motifs, and predicted disorder.The corpus is curated from the AlphaFold Database.Source: §1 · p. 2
  • RL-SAE rewards sequences that activate specified sets of interpretable SAE features, averaging 90% activation of 30 targeted features across eight tasks versus 24% for activation steering.This provides explicit control over function-associated sequence patterns.Source: §1 · p. 2
  • RL-SAE improves predicted subcellular localization and transcriptional activity over activation steering and supervised fine-tuning while enabling function-associated features to be incorporated and combined selectively.Source: §1 · p. 2

2 RELATED WORK

Prior IDR design approaches impose disorder through hand-specified rules, while protein language model steering does not ensure that multiple target features activate together. IDiom learns IDR sequence statistics, and RL-SAE rewards specified feature sets.

2 RELATED WORK

  • Residue-sampling approaches impose disorder through hand-specified constraints and may not capture the evolutionary statistics of natural IDRs.Source: §2 · p. 2
  • Combining activation-steering directions does not ensure that generated sequences activate all targeted features; RL-SAE activates a greater fraction and supports combining features associated with distinct functions.Source: §2 · p. 2
  • Unlike related SAE approaches that select or amplify feature directions or fit rewards to preference pairs, RL-SAE directly rewards activation of prespecified feature sets.Source: §2 · p. 3

3 GENERATIVE MODELING OF INTRINSICALLY DISORDERED REGIONS

IDiom-DB supplies a large predicted-IDR corpus, and IDiom is trained to generate sequences with or without flanking context. Generated sequences show novelty and closely match natural IDRs on predicted disorder, composition, and patterning.

3 GENERATIVE MODELING OF INTRINSICALLY DISORDERED REGIONS

  • IDiom-DB contains 54M IDRs identified from AlphaFold Database predictions using pLDDT thresholds and sequence filters.The curation removes regions shorter than 30 residues, proteins longer than 1,020 residues, entirely low-pLDDT proteins, and predicted N-terminal signal peptides.Source: §3 · p. 3
  • IDiom is a decoder-only transformer trained with a next-token objective on fill-in-the-middle sequences, supporting prompted and unprompted generation.Half of training sequences include flanking contexts and half do not; analyses focus on the 302M-parameter model.Source: §3 · p. 3
  • Generated sequences’ maximum identity to training IDRs peaks broadly around 50%, indicating most are substantially dissimilar to training sequences.Source: §3 · p. 4
  • Generated IDRs closely track DisProt in predicted disorder, composition, and patterning, and have higher predicted disorder scores than folded CATH domains.Reported patterning properties include charge segregation, reduced hydrophobic clustering, and low sequence complexity.Source: §3 · p. 4

4 REINFORCEMENT LEARNING WITH SPARSE AUTOENCODER FEATURES

RL-SAE addresses the challenge of inducing many interpretable sequence features simultaneously by selecting features enriched in labeled sequences and rewarding their activation during post-training. Its reward also penalizes deviations from target length and composition entropy.

4 REINFORCEMENT LEARNING WITH SPARSE AUTOENCODER FEATURES

  • RL-SAE targets SAE features enriched in sequences with common labeled functionality, aiming to induce multiple function-associated patterns simultaneously.Source: §4 · p. 4
  • IDiomSAE is a Top-k sparse autoencoder with 16,384 latents, trained on IDR-residue activations from IDiom’s layer 18 of 24.For each residual-stream activation, its encoder produces sparse features and its decoder reconstructs the activation.Source: §4 · p. 4
  • Features are ranked by the log odds ratio of firing prevalence in positive versus negative labeled sets, and the top-N are selected as the target set.A feature counts as firing if it activates at any residue position in a sequence.Source: §4 · p. 4
  • The reward favors the fraction of target features that fire anywhere in a generated sequence, with quadratic penalties for relative deviations from target length and amino-acid composition entropy.RL post-training uses group-relative policy optimization with KL regularization relative to the base IDiom model.Source: §4 · pp. 4–5

5 EXPERIMENTS

Across compartment-association and transcriptional-regulation design tasks, the experiments test whether RL-SAE can install interpretable IDR features and improve predicted function. They also examine how private-feature selection and feature compatibility affect specificity and the organization of composed sequences.

5.1 EXPERIMENTAL SETUP

  • The experiments use annotated human compartment and effector-domain sequence sets, generate unprompted 80- or 100-residue IDRs, and compare RL-SAE with steering and supervised fine-tuning.Source: §5.1 · p. 5

5.2.1 ENRICHED FEATURES RECOVER CANONICAL IDR SEQUENCE GRAMMARS

  • Enriched features recover recognizable grammars, including FG/GLFG repeats, RS repeats, acidic activation-domain patterning, and SUMOylation motifs, providing interpretable targets for RL-SAE.Source: §5.2.1 · p. 6

5.2.2 DESIGNING BIOMOLECULAR CONDENSATE-ASSOCIATED IDRS

  • RL-SAE sequences activate 90% of targeted top-30 features on average, versus 24% for activation steering, 15% for SFT, and 21% for natural sequences.Source: §5.2.2 · p. 6
  • For all compartments except P-bodies, top-30 RL-SAE designs have higher mean predicted localization scores than SFT, steering, and natural IDRs; scrambles indicate grammar-driven scores except partly for stress granules and P-bodies.Source: §5.2.2 · p. 6
  • Private-30 targeting reduces selected off-target ProtGPS scores while retaining higher mean on-target scores than SFT, steering, and natural IDRs across all six compartments, though some off-target pairs increase.Source: §5.2.2 · p. 7

5.2.3 DESIGNING AND COMPOSING TRANSCRIPTIONAL EFFECTOR DOMAINS

  • RL-SAE designs reach mean PADDLE scores approximately 209% of natural ADs and generate over 5 SUMOylation motifs per sequence for repression.Source: §5.2.3 · p. 8
  • Composed feature organization differs by compatibility: chromosome and AD features occupy minimally overlapping residues and form distinct charge blocks, while chromosome and RD features overlap more and have lower κ.Source: §5.2.3 · p. 9, Figure 4 · p. 9
  • Chromosome + AD/RD designs install an average of 53 of 60 target features, while retaining predicted activation or repression function alongside chromosome association and nuclear localization.Source: §5.2.3 · pp. 8–9, Figure 4 · p. 9

6 CONCLUSIONS

IDiom and RL-SAE provide a framework for generating IDRs that recapitulate natural sequence properties and combine interpretable features. The evaluations are computational, and experimental validation is needed to establish whether predicted properties translate into biological function.

6 CONCLUSIONS

  • IDiom generates diverse IDR sequences that recapitulate natural IDR properties, while RL-SAE rewards generation of sequences activating specified SAE feature sets.The framework is presented for identifying and composing interpretable sequence features in IDR design.Source: §6 · p. 10
  • Computational experiments show RL-SAE installs tens of target features and produces sequences with strong predicted biological function.The authors state that future experimental validation could establish the utility of these methods for engineering biological function.Source: §6 · p. 10
  • RL-SAE requires training, and scanning many combinations of feature sets could be computationally intensive.Top-N feature selection may omit functionally relevant features or include redundant ones; experimental validation is needed to confirm predicted properties translate into biological function.Source: §6 · p. 10

ETHICS STATEMENT

The work develops a computational method for generative modeling of IDRs, with generated sequences requiring validation and biosafety assessment before future use.

ETHICS STATEMENT

  • Generated sequences must undergo appropriate experimental validation and biosafety assessment before future use.The evaluations in this work are computational.Source: ETHICS STATEMENT · p. 10
  • The authors note that protein sequence generation methods may be repurposed for harmful use, which should be considered when distributing and applying them.Source: ETHICS STATEMENT · p. 10
  • The authors state that they comply with the licenses and attribution requirements of all datasets used.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The paper reports AI assistance in literature search, editing, and research-code implementation, with revisions and manual verification. IDiom-DB construction curates predicted IDRs from AlphaFold Database sequences and prepares them for model training.

AI USE STATEMENT

  • Generative AI tools assisted with literature searches, manuscript editing, and research-code implementation; the authors revised edited text and manually verified implemented code.The authors take responsibility for the final content, including AI-assisted text, claims, or artifacts.Source: AI USE STATEMENT · p. 10

A IDIOM-DB CONSTRUCTION

  • Figure 5 characterizes full-length and IDR lengths, the IDR fraction of full-length proteins, and entropy distributions for IDR versus non-IDR regions.The red dashed line marks the lower-5th-percentile entropy threshold.Source: Figure 5 · p. 19
  • IDiom-DB curation clusters 214M AlphaFold Database sequences, identifies pLDDT-based IDRs, and yields 54M IDRs with flanking contexts after filtering.Filters exclude proteins longer than 1,020 residues, IDRs shorter than 30 residues, and sequences entirely low-pLDDT; signal-peptide filtering removes 3.6M records.Source: A IDIOM-DB CONSTRUCTION · pp. 19–20
  • Generated sequences are compared with IDiom-DB IDRs using the closest-match sequence identity, whose distribution is plotted in Figure 1f.The search is performed separately for prompted and unprompted generated IDRs against the 54M training IDRs.Source: A IDIOM-DB CONSTRUCTION · p. 19
  • To limit benchmark leakage, the training set removes IDRs aligning to experimentally annotated DisProt IDRs at 50% identity and 80% bidirectional coverage.The retained reference set contains 1,663 benchmark IDRs, and the procedure removes 55K training records.Source: A IDIOM-DB CONSTRUCTION · pp. 19–20
  • The 27-token alphabet marks N- and C-terminal contexts, the IDR span, sequence boundaries, padding, and masking.Sequences are split 99%/0.5%/0.5% into training, validation, and test sets and transformed into fill-in-the-middle order; training presents flanks with 50% probability.Source: A IDIOM-DB CONSTRUCTION · p. 20

B MODEL ARCHITECTURE, PRETRAINING, AND INFERENCE

IDiom is a decoder-only transformer pretrained autoregressively on IDiom-DB. Its inference procedure samples tokens autoregressively and supports generation with or without flanking-context prompts.

B MODEL ARCHITECTURE, PRETRAINING, AND INFERENCE

  • IDiom is a decoder-only transformer trained at 19M, 85M, and 302M parameter sizes.The architecture uses RMSNorm, query-key normalization, SwiGLU, RoPE, and a maximum sequence length of 1,024 tokens.Source: B MODEL ARCHITECTURE, PRETRAINING, AND… · p. 21
  • IDiom is pretrained on IDiom-DB with next-token prediction, using 1,024-token padded sequences.Training uses AdamW without weight decay for 250,000 optimizer steps, with a batch size of 1,024 and model-size-specific learning rates.Source: B MODEL ARCHITECTURE, PRETRAINING, AND… · p. 21
  • Inference samples each next token from the temperature-scaled model distribution at T = 1.0 and stops at EOS or 1,024 tokens.Prompted generation uses N- and C-terminal flanks before the IDR marker; unprompted generation uses only the special context and IDR tokens.Source: B MODEL ARCHITECTURE, PRETRAINING, AND… · p. 21

C CHARACTERIZATION OF GENERATED SEQUENCES

IDiom-generated IDRs reproduce several sequence properties observed in natural IDRs, including composition, charge patterning, hydrophobic patterning, and entropy. Prompted sequences show the closest compositional agreement with DisProt, while both prompted and unprompted sequences match its entropy distribution well.

Amino acid composition

  • Generated IDR compositions closely match natural DisProt sequences across most amino acids, with the closest agreement in sequences prompted by DisProt flanking contexts.Differences in cysteine and charged-residue composition may reflect DisProt’s small size and selection bias.Source: C CHARACTERIZATION OF GENERATED… · p. 22

Charge content

  • Generated and natural IDRs span a wider range of FCR values than folded CATH domains, reflecting the heterogeneity of IDR charge content.The range includes highly charged sequences and weakly charged low-complexity sequences.Source: C CHARACTERIZATION OF GENERATED… · p. 23

Charge patterning

  • IDiom reproduces the high-κ tail of natural IDRs relative to CATH domains, consistent with charge segregation in some disordered sequences.κ near 0 indicates well-mixed opposite charges, while κ near 1 indicates segregation into same-charge blocks.Source: C CHARACTERIZATION OF GENERATED… · p. 23

Hydrophobic patterning and sequence entropy

  • Generated IDRs show lower SHD than folded CATH domains, while closely matching the training and DisProt IDR trends.Prompted and unprompted IDRs also reproduce the lower entropy of natural IDRs relative to folded domains, with generated entropy matching the DisProt distribution well.Source: C CHARACTERIZATION OF GENERATED… · p. 23

D SPARSE AUTOENCODER TRAINING AND CHARACTERIZATION

The paper trains IDiomSAEs on IDiom’s IDR activations, selects enriched features from labeled sequence sets, and uses RL-SAE to reward their activation. It compares RL-SAE with steering, supervised fine-tuning, and an oracle-reward baseline, reporting results across three-seed runs.

D SPARSE AUTOENCODER TRAINING AND CHARACTERIZATION

E FEATURE ENRICHMENT PROCEDURE

  • Enriched features are filtered by edge proximity, false-discovery rate, log odds ratio, and positive-set firing frequency, then ranked by log odds ratio.Main-text analyses use the top 30 features.Source: E FEATURE ENRICHMENT PROCEDURE · p. 25

F COMPARTMENT AND EFFECTOR DOMAIN SEQUENCES

  • Compartment positives are IDRs from six localization groups; effector-domain positives are screened activation or repression tiles, with length-matched IDiom-DB IDRs or inert tiles as negatives.Compartment IDRs are 30–1,020 residues; effector positives comprise 288 activation-domain and 408 repression-domain sequences.Source: F COMPARTMENT AND EFFECTOR DOMAIN… · p. 26

G RL-SAE DETAILS

  • RL-SAE uses GRPO rewards based on frozen IDiom and IDiomSAE feature activations, and scores generally trend upward as more target features are included.Top-30 is used because it is the largest tested N for which all six compartments have sufficient private features; ProtGPS scores show substantial variability.Source: G RL-SAE DETAILS · p. 27, Figure 8 · p. 27, Figure 9 · p. 27

H BASELINE METHODS

  • Activation steering adds a normalized sum of target-feature decoder directions to the layer-18 residual stream, while supervised fine-tuning trains on each target’s positive set.Steering sweeps α and excludes sequences below the IDiom-DB fifth-percentile entropy threshold; RL-GPS instead uses ProtGPS scores as rewards for compartment tasks.Source: H BASELINE METHODS · p. 28
  • The oracle RL baseline replaces the RL-SAE feature reward with the ProtGPS score for the target compartment.Its other hyperparameters match RL-SAE.Source: H BASELINE METHODS · p. 28

I REPORTED UNCERTAINTIES

  • Reported method comparisons average scores across three seeds; DeepLoc predictions use 200 sequences per run, while other metrics use 1,000.Natural IDR-set means are reported with standard errors.Source: I REPORTED UNCERTAINTIES · p. 29

J TOP-N COVERAGE

Across the top-N coverage analyses, RL-SAE attains near-complete target-feature activation in most tested cases, whereas steering often loses coverage as more directions are combined. Additional controls describe outcomes for random targets and identify the comparisons reported for oracle RL and composition-matched scrambles.

J TOP-N COVERAGE

  • RL-SAE reaches RS values near 1.0 across most compartments and N values, while steering coverage drops with increasing N in five of eight sequence sets.RL-SAE has exceptions at several low-N cases; steering’s plotted values use its best α even when that α fails the entropy threshold, making them an upper bound.Source: Figure 10 · p. 30, J TOP-N COVERAGE · p. 30
  • Steering scores commonly peak at α values of 0.2–0.45 and then decline as sequence entropy falls; in four sets, the peak and decline coincide with crossing the entropy cutoff.The authors hypothesize the score drop may coincide with sequences leaving the natural IDR entropy range.Source: Figure 11 · p. 31, Chromosome · p. 31

K RANDOM FEATURE SELECTION CONTROL

L ORACLE RL BASELINE (RL-GPS)

  • Table 5 reports ProtGPS-predicted compartment association scores for the RL-GPS oracle baseline, with standard deviations across three training seeds.Source: Table 5 · p. 33

M COMPOSITION SCRAMBLE CONTROLS

  • Tables 6 and 7 compare RL-SAE sequences with composition-matched scrambles for localization and transcriptional-effector readouts, respectively.Source: Table 6 · p. 34, Table 7 · p. 34

N ADDITIONAL DISORDER PREDICTIONS

This section presents additional disorder predictions for designed sequences alongside sequence examples spanning subcellular compartments and activation or repression domains.

N ADDITIONAL DISORDER PREDICTIONS

  • Both Metapredict and IUPred classify a sequence as disordered when its score is greater than 0.5.The reported disorder predictions use Metapredict V3 and IUPred3.Source: N ADDITIONAL DISORDER PREDICTIONS · p. 35
  • Metapredict predicted disorder across methods and sequence sets except RL-GPS NPC sequences, which scored 0.44; the authors suggest this may reflect hacking of the ProtGPS reward.RL-GPS on NPC nevertheless achieved a high ProtGPS score.Source: N ADDITIONAL DISORDER PREDICTIONS · p. 35
  • IUPred scores were mostly near or above 0.5, except activation-steered sequences (0.39) and RL-GPS NPC sequences (0.12); the authors link these results to hydrophobic content and possible reward hacking, respectively.The authors note activation-steered sequences may remain functionally relevant to co-activator recruitment despite the low IUPred score.Source: N ADDITIONAL DISORDER PREDICTIONS · p. 35

Natural nuclear speckle IDRs

RL-GPS, Stress granule

Natural stress granule IDRs

RL-GPS, P-body

Natural P-body IDRs

RL-GPS, Nuclear pore complex

Natural nuclear pore complex IDRs

Natural activation domains

Natural repression domains

Loading 2610.02189v1…