PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Tokens: Probing Higher-Order Epistasis in Learned Protein Representations

Maryam Rahimimovassagh, Ivan Garibay, Niloofar Yousefi

arXiv:2608.24953v1q-bio.QMcs.LG

TL;DR

Protein fitness landscapes contain higher-order epistasis, motivating methods that reveal where interaction information is represented and recovered. ORBIT combines Walsh-based synthetic validation with representation and function analysis on GB1 under FLIP 2-vs-rest. The primary comparison found early pairwise accessibility gains for RIT but no downstream higher-order advantage, while deeper MLPs improved several higher-order measures.

  • Problem

    Higher-order epistasis makes mutation effects context-dependent, while similar predictive performance can conceal different internal organization and accessibility of interaction information.

  • Method

    ORBIT measures interaction presence, representation accessibility, and functional recovery using Walsh-based diagnostics across synthetic landscapes and GB1 models.

  • Results

    RIT increased pairwise token-stage accessibility without detectable downstream higher-order or FLIP prediction gains, whereas additional MLP depth/capacity improved third-order recovery and higher-order accessibility.

  • Takeaways & Limitations

    Early interaction-aware encoding and later nonlinear construction are distinct routes by which interaction information can become accessible.

  • Takeaways & Limitations

    The depth experiment changes both network depth and parameter count, so its improvements establish depth/capacity sensitivity rather than a parameter-matched causal effect of depth.

Abstract

from arXiv · show

Protein fitness landscapes contain nonlinear interactions in which mutation effects depend on other residues. We introduce ORBIT, an Order-Resolved Benchmarking of Interaction Transformations framework that separates interaction presence, representation accessibility, and functional recovery. ORBIT first validates Walsh-based diagnostics on synthetic landscapes with known interaction order, then analyzes the experimentally measured GB1 fitness landscape under the FLIP 2-vs-rest setting. We compare ridge regression, a standard MLP, independent tokens, nonlinear independent tokens, and Residual Interaction Tokenization (RIT). Across 20 paired training seeds, the primary two-hidden-layer comparison found no significant architecture differences in FLIP test R^2, third- or fourth-order functional recovery, or final-layer third- or fourth-order accessibility. However, RIT significantly increased pairwise accessibility at the token stage relative to both independent-token controls (Delta A_tok,2 = 0.2468, d_z = 1.67, Holm-adjusted p = 1.14 x 10^-5), without a detectable downstream higher-order advantage. A pre-specified depth/capacity analysis showed that deeper MLPs improved FLIP prediction, third-order functional recovery, and final-layer third-order accessibility; fourth-order accessibility also improved relative to the shallow MLP but remained below zero in absolute held-out R^2. ORBIT therefore reveals representation-level changes hidden by conventional prediction metrics and distinguishes early interaction-aware encoding from higher-order structure constructed by downstream nonlinear capacity.

1 Introduction

Protein fitness landscapes exhibit higher-order epistasis, but predictive accuracy alone does not reveal how models organize or access interaction structure. ORBIT separates interaction presence, representation accessibility, and functional recovery across synthetic and GB1 benchmarks.

  • Higher-order epistasis arises when mutation effects depend on other residues and cannot be explained by independent single-mutation effects.
  • GB1 provides a compact, experimentally measured landscape for testing generalization from lower-mutation variants to triple and quadruple mutants.The benchmark measured 149,361 of 160,000 combinations across four epistatic positions.
  • ORBIT distinguishes interaction presence, representation accessibility, and functional recovery as separate levels of analysis.It quantifies these levels using representation Walsh energy, held-out probe performance, and order-specific functional R^2.
  • RIT increased pairwise accessibility at the token stage without detectable downstream advantages in higher-order recovery, final-layer accessibility, or FLIP test R^2.
  • Additional downstream capacity improved third-order functional recovery and higher-order accessibility for the MLP, providing a route distinct from early interaction-aware encoding.
  • ORBIT extends spectral analysis from learned functions to intermediate representations, asking where order-specific information becomes accessible.

3 ORBIT: Order-Resolved Representation Analysis

ORBIT applies Walsh decompositions to both predicted fitness landscapes and internal representations. Its separate energy and held-out accessibility measures distinguish order-specific variation from experimentally aligned, linearly recoverable structure.

  • Walsh expansions assign interaction order to coefficients according to the cardinality of their indexed variable subset.For four GB1 positions, the decomposition contains first- through fourth-order components.
  • Functional recovery measures how faithfully predicted fitness landscapes reproduce experimentally measured Walsh coefficients at each interaction order.
  • ORBIT extends the same decomposition to intermediate representations, whose vector-valued Walsh coefficients describe variation associated with indexed Walsh patterns.These coefficients are not interpreted as biochemical interaction strengths.
  • Normalized representation Walsh energy measures the fraction of nonconstant representation energy at each order, indicating interaction presence rather than recoverability.
  • Accessibility is measured by probing representation components to predict corresponding experimental Walsh coefficients on held-out data.A_l,k therefore measures linear accessibility of experimentally measured order-k structure from stage l.
  • The AA-identity-strict protocol holds out non-WT amino-acid identities by position, while probe-test sequences may have appeared in upstream model training.

4 ORBIT-Synth

ORBIT-Synth validates order-resolved Walsh diagnostics under controlled synthetic ground truth before applying them to complete local GB1 landscapes. The GB1 construction supports matched function- and representation-level analysis under FLIP generalization.

  • ORBIT-Synth: Pure-order controls place nonzero Walsh coefficients at only one interaction order, making the normalized spectrum an exact ground-truth test.
  • ORBIT-Synth: The parity control assigns a nonzero coefficient only to the full variable subset, producing an exact full-order parity interaction under binary coding.
  • ORBIT-Synth: The cooperativity sweep varies the relative contribution of lower- and higher-order structure using disjoint unit-energy Walsh coefficient vectors.The same sampled coefficient directions are reused across conditions.
  • ORBIT-Synth: ORBIT-Synth uses noiseless synthetic landscapes with known pure first- through fourth-order structure, exact fourth-order parity, and a cooperativity sweep.
  • GB1: GB1 varies four B1-domain positions and contains substantial epistatic structure while remaining small enough for exact four-site analysis.The measured landscape contains 149,361 variants.
  • GB1: The FLIP 2-vs-rest experiment trains on wild type, single mutants, and double mutants, then tests on triple and quadruple mutants.This tests generalization from lower-mutation-complexity variants to higher-mutation combinations.
  • GB1: Complete WT-anchored four-dimensional binary cubes provide shared objects for function-level Walsh decomposition and representation-level ORBIT analysis.

6 Models

The study uses lightweight, matched models to isolate how token construction and downstream nonlinear capacity organize interaction information. RIT adds pairwise cross-position information at the token stage, while depth analysis tests sensitivity to additional downstream capacity.

  • Model set: Five predictors are evaluated: ridge regression, a standard MLP, independent tokens, nonlinear independent tokens, and Residual Interaction Tokenization (RIT).Ridge provides a linear additive reference, while the neural architectures differ in representation construction and downstream processing.
  • Model set: The standard MLP constructs cross-position interaction structure through two downstream nonlinear hidden layers rather than explicit interaction-aware tokenization.It receives concatenated one-hot sequences and is analyzed at hidden stages H1 and H2.
  • Token architectures: Independent-token models create each token from only its own residue identity, leaving Ztok without an explicit cross-position interaction mechanism.The nonlinear independent-token control adds token-wise nonlinearity while preserving this independence.
  • Token architectures: RIT introduces pairwise cross-position information before the shared predictor through interaction-aware residual tokens.Its learnable residual coefficient starts at 0.05, keeping the initial representation close to independent tokens while allowing interaction-dependent corrections.
  • Experimental controls: The primary token architectures match token dimensions, hidden dimensions, optimizer, training budget, data partitions, and the two-hidden-layer backbone.Their principal difference is how Ztok is constructed before entering the common predictor.
  • Experimental controls: The depth/capacity analysis compares neural backbones with two, three, and four hidden layers using the same 20 paired training seeds.Because deeper networks add parameters, this analysis does not isolate a causal effect of depth under parameter-matched capacity.

7 Evaluation and Statistical Inference

Evaluation separates prediction, functional recovery, and representation-level interaction measurements rather than collapsing them into one score. Confirmatory inference uses paired neural-model comparisons across shared seeds with prespecified endpoint families and multiplicity correction.

  • Evaluation measures: The evaluation combines FLIP prediction, order-specific functional recovery, and representation accessibility measurements.These metrics assess complementary aspects of generalization and learned interaction structure.
  • Evaluation measures: FLIP test R2 is the pre-specified confirmatory prediction endpoint, with RMSE and Spearman correlation reported descriptively.The test evaluates generalization from training variants with HD ≤2 to experimentally measured triple and quadruple mutants.
  • Representation measures: Representation energy Eℓ,k measures the fraction of nonconstant representation Walsh energy associated with interaction order k.It remains descriptive, without heatmap-wide inferential testing.
  • Representation measures: Accessibility Aℓ,k is held-out ridge-probe R2 for predicting experimentally measured order-k Walsh coefficients from representation-level Walsh components.Functional recovery separately measures agreement between experimentally measured and predicted order-specific Walsh coefficients.
  • Confirmatory endpoints: The confirmatory higher-order endpoints are third- and fourth-order functional recovery and final-layer accessibility, with token-stage pairwise accessibility tested for RIT.Intermediate-layer trajectories are descriptive unless they correspond to prespecified endpoints.
  • Statistical inference: Neural-model inference uses 20 paired training seeds, exact paired sign-flip permutation tests, bootstrap intervals, standardized effect sizes dz, and Holm-adjusted p-values.Holm adjustment is applied separately across prediction, higher-order recovery, final-layer accessibility, and token-stage pairwise-accessibility families.

8 Experimental Questions

ORBIT addresses whether Walsh diagnostics recover known interaction order, whether low-complexity training generalizes to higher-order mutants, where interaction information becomes accessible, and whether prediction can diverge from internal organization. A prespecified depth analysis tests sensitivity to additional downstream capacity.

  • Synthetic validation: ORBIT is first validated on synthetic landscapes with known first- through fourth-order structure, including exact fourth-order parity.The validation includes a controlled transition from lower- to higher-order Walsh energy.
  • Generalization: Under FLIP 2-vs-rest, models train on wild type, single mutants, and double mutants and are evaluated on experimentally measured triple and quadruple mutants.This tests generalization from lower-mutation-complexity variants to unseen higher-mutation combinations.
  • Representation accessibility: ORBIT measures where order-specific information becomes accessible across learned representation stages for interaction orders k = 1, . . . , 4.The analysis uses normalized representation Walsh energy and related accessibility measurements.
  • Token-stage comparison: The token-stage test compares Independent tokens, Nonlinear independent tokens, and RIT on pairwise accessibility A_tok,2.The central question is whether interaction-aware tokenization changes where pairwise information first becomes accessible.
  • Cross-metric comparison: ORBIT compares conventional fitness prediction with order-specific accessibility and functional recovery to test whether similar final predictions can coexist with different internal organization.This directly addresses divergence between predictive performance and representation-level structure.

9 Results

ORBIT validated its order-resolved diagnostics on synthetic landscapes and exposed differences between prediction, functional recovery, and internal accessibility in GB1. RIT improved pairwise accessibility early, while deeper MLPs improved several higher-order outcomes without producing uniform architecture advantages.

  • ORBIT-Synth validation: Synthetic benchmarks recovered the planted interaction order, including the intended first- to fourth-order transition in the cooperativity sweep.Increasing C from 0 to 1 produced the prescribed progression from purely first-order to purely fourth-order structure.
  • GB1 prediction: All nonlinear neural models generalized beyond the FLIP training mutation complexity, but no architecture was statistically dominant in the primary two-hidden-layer comparison.Independent tokens had the highest mean FLIP test R2, Nonlinear tokens the highest mean Spearman correlation, and RIT remained competitive.
  • Functional recovery: Functional recovery declined with interaction order, with nonlinear models outperforming ridge at first and second order but becoming weaker and more variable at third and fourth order.Descriptive model rankings at higher orders were not supported by significant architecture-level differences after Holm correction.
  • Representation presence and accessibility: RIT introduced second-order representation energy at the token stage and substantially increased pairwise accessibility relative to independent-token controls.Later layers redistributed some energy toward higher orders, but the early pairwise accessibility advantage did not extend to significant final-layer third- or fourth-order accessibility.
  • Depth/capacity sensitivity: Deeper standard MLPs improved FLIP prediction, third-order functional recovery, and final-layer third-order accessibility relative to shallower networks.Fourth-order accessibility also improved, although no token-based architecture showed a significant confirmatory depth effect on final-layer A3 or A4.
  • Interpretation: The results distinguish early interaction-aware encoding by RIT from later higher-order construction enabled by downstream nonlinear MLP capacity.Conventional fitness prediction alone does not determine how interaction information is organized internally.

10 Discussion

ORBIT separates interaction presence, linear accessibility, and functional recovery, revealing that similar predictive performance can arise from different representation trajectories. RIT improved early pairwise accessibility, whereas deeper MLPs improved several higher-order outcomes without establishing a parameter-matched causal effect of depth.

  • Interaction trajectories: ORBIT distinguishes whether interaction structure is present, linearly accessible, and recovered in the final predicted fitness landscape.Walsh-based diagnostics were validated on synthetic landscapes before application to experimental GB1 data.
  • Primary comparison: The primary matched-backbone comparison found no statistical evidence that token architectures differed in FLIP test R^2.Independent tokens had the highest mean FLIP test R^2, while Nonlinear tokens had the highest mean Spearman correlation, but these descriptive differences were not statistically supported.
  • Primary comparison: None of the pre-specified architecture contrasts for third- or fourth-order functional recovery remained significant after multiplicity correction.Descriptive differences therefore do not establish that one tokenization strategy reliably improves higher-order recovery.
  • Interaction trajectories: RIT increased token-stage pairwise accessibility relative to both independent-token controls.The paired comparison found ΔA_tok,2 = 0.2468, d_z = 1.67, and Holm-adjusted p = 1.14 × 10^-5.
  • Interaction trajectories: RIT’s early accessibility advantage did not yield detectable third- or fourth-order gains in final-layer accessibility, functional recovery, or FLIP prediction.Thus, earlier access to interaction information did not necessarily change the downstream function reconstructed by the predictor.
  • Depth and capacity: Deeper MLPs significantly improved FLIP prediction, third-order functional recovery, and final-layer third-order accessibility.Fourth-order accessibility also improved relative to the two-layer MLP, although absolute A4 remained below zero.
  • Interpretation: Together, the results support distinct routes of early interaction encoding and later nonlinear interaction construction.The deeper MLP provided evidence for downstream construction even without an explicit initial interaction mechanism.
  • Depth and capacity: The depth experiment establishes depth/capacity sensitivity rather than a parameter-matched causal effect of depth.Three- and four-layer models contained more parameters than their two-layer counterparts.

11 Limitations and Future Work

The study’s conclusions are bounded by the compact GB1 setting, the interpretation of Walsh coefficients, probe choice, split design, and confounding of depth with parameter count. Future work extends ORBIT to broader landscapes and modern cross-token processors.

  • Scope: GB1 varies only four amino-acid positions, making it a compact epistasis system rather than a model of full protein sequence–function complexity.The authors propose evaluating ORBIT on larger combinatorial landscapes and additional molecular domains.
  • Interpretation: Walsh coefficients quantify statistical epistasis and should not automatically be interpreted as physical contacts, biochemical mechanisms, or causal residue relationships.
  • Probe design: Accessibility results depend on the fixed ridge probe and therefore measure linear accessibility under a common decoding rule.A nonlinear probe could recover additional information but would address decodability rather than linear accessibility.
  • Probe design: The AA-identity-strict protocol holds out non-WT identities at each position but is not an upstream sequence-disjoint model-training split.WT-anchored cubes can share the WT sequence and overlap in other sequence content.
  • Depth analysis: The depth experiment changes both network depth and parameter count, so its findings establish depth/capacity sensitivity rather than a causal effect of depth alone.A parameter-matched follow-up is required to separate these contributions.
  • Future work: The lightweight architectures make representation trajectories auditable but do not establish behavior under modern cross-token processors.Transformer-style self-attention is proposed as a next step for tracking interaction structure across attention layers.
  • Conclusion: ORBIT is presented as an order-resolved framework for tracing nonlinear interaction structure through learned protein representations.It combines synthetic validation, GB1 generalization, representation-level Walsh analysis, and order-specific functional recovery.

A Reproducibility and Frozen Experimental Protocol

The reproducibility protocol freezes data construction, evaluation order, stochastic replication, statistical analysis, and provenance controls for the ORBIT experiments. It documents the GB1 and FLIP partitions, strict probe split, and clean rerun before reporting.

  • Protocol scope: The appendix records the frozen experimental and inferential protocol underlying the reported ORBIT results.It covers data artifacts, execution order, training seeds, depth/capacity analysis, probe fitting, statistics, and provenance controls.
  • Data and splits: The GB1 dataset contains 149,361 experimentally measured variants across four positions.The positions are V39, D40, G41, and V54.
  • Data and splits: The official FLIP 2-vs-rest artifact contains 424 training variants and 8,309 test variants.
  • Data and splits: The prediction-training partition contains wild type, single mutants, and double mutants, while testing uses triple and quadruple mutants.The corresponding Hamming-distance ranges are HD ≤2 for training and HD = 3–4 for testing.
  • Data and splits: The AA-identity-strict probe construction produced 7,352 training cubes, 245 validation cubes, and 613 test cubes.
  • Data and splits: Non-WT amino-acid identities are disjoint across probe partitions at each position, while the WT identity is intentionally shared.
  • Execution order: The frozen pipeline validates Walsh and ORBIT-Synth tests, prepares GB1, freezes splits, constructs cubes, trains models, and computes prediction and recovery analyses.It then runs pre-specified statistical tests and generates manuscript and statistical-summary outputs.
  • Execution order: The complete primary pipeline was rerun from clean archived outputs before final reporting.

A.3 Stochastic Replication

The primary GB1 analysis used paired stochastic seeds and held-out probes to compare interaction recovery, accessibility, and representation energy across model stages. Functional recovery weakened at higher orders, while RIT showed an early pairwise-accessibility advantage that did not translate into a final higher-order advantage.

  • Experimental design: Neural-model results summarize 20 stochastic training seeds, with functional recovery evaluated across interaction orders 1–4 and representation probes applied at model-specific stages.Token-based models were assessed at Ztok, H1, and H2; the MLP was assessed at H1 and H2.
  • Functional recovery: Higher-order functional recovery was weaker and more variable than first- and second-order recovery, despite nonlinear models outperforming Ridge at lower orders.Descriptive neural-architecture differences at higher orders did not support a reliable architecture-level advantage in the pre-specified comparisons.
  • Representation accessibility: RIT made pairwise epistatic information substantially more accessible at the token stage than either independent-token control, but this advantage did not persist as a significant final-layer third- or fourth-order effect.The result distinguishes early accessibility from downstream functional recovery.
  • Representation energy: First-order representation energy dominated most stages; independent-token controls began with only first-order structure, whereas RIT introduced second-order energy through explicit pairwise interactions.Subsequent nonlinear layers redistributed some energy toward higher orders, but third- and fourth-order components remained comparatively small.
  • Interpretation: ORBIT separately measures order-specific variation, experimental alignment, and final recovery because presence alone does not establish accessibility or improved predicted fitness.Representation Walsh energy is descriptive, while held-out accessibility and order-specific functional recovery address distinct questions.

C Complete Depth/Capacity Results

The depth/capacity experiment compared MLP, independent-token, nonlinear-independent-token, and RIT models at two, three, and four hidden layers using paired seeds. Because deeper models added parameters, the results measure sensitivity to increased depth and capacity rather than depth alone.

  • Experiment: The depth experiment evaluated four architectures at depths 2, 3, and 4 using the same 20 paired training seeds as the primary analysis.The reported views include FLIP prediction, functional Walsh recovery, layer- and order-resolved accessibility, and representation Walsh energy.
  • Scope: Deeper models add 32-dimensional hidden layers after H2, so the experiment tests depth/capacity sensitivity rather than a parameter-matched causal effect of depth.This scope boundary applies to interpreting the deeper-backbone comparisons.
  • Statistical analysis: The confirmatory analysis covered 57 comparisons across higher-order functional recovery, final-layer higher-order accessibility, and token-stage pairwise accessibility families.Holm correction was performed separately within the pre-specified multiplicity families.
  • Inference boundaries: Representation-energy trajectories and intermediate-layer accessibility were descriptive, while confirmatory inference was restricted to prediction, higher-order recovery, and final-layer accessibility endpoints.All pre-specified comparisons were reported regardless of significance or effect direction.
  • Confirmatory depth contrasts: Final-layer fourth-order accessibility for the MLP improved by 0.165 for d3 − d2, with a 95% interval of [0.084, 0.262] and Holm-adjusted p = 0.01.This was a pre-specified depth contrast; d4 − d3 was not confirmatory.
Loading 2608.24953v1…