PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling

Yifan Feng, Guanjie Cheng, Shihui Ying, Shaoyi Du, Yue Gao

arXiv:2608.29207v2cs.AIcs.LG

TL;DR

Protein structure modeling needs expressive interactions between residue content and three-dimensional geometry, but the expressive limit of such layers was not systematically established. The paper identifies the complete bilinear operator as that ceiling and introduces Hyper-Fold, whose rank-K matrix gating approaches it efficiently; across structure-encoder tasks, it reports strong results, including structure-only pocket detection that surpasses UniSite-3D.

  • Problem

    The paper asks what expressive limit governs sequence–geometry interaction layers and where existing models lie relative to that limit.

  • Method

    Hyper-Fold uses rank-K separable, geometry-conditioned matrix operators and organizes each residue neighborhood into sequence and contact hyperedges.

  • Results

    Across enzyme function prediction, fold classification, and pocket detection, Hyper-Fold variants achieve strong structure-encoder results; Hyper-Fold-Pocket reaches 0.617 AP@0.3 versus 0.560 for UniSite-3D on UniSite-DS.

  • Takeaways & Limitations

    A sufficiently expressive 3D backbone can recover information that fusion architectures previously borrowed from evolution-scale pretraining, without sequence language-model features.

  • Takeaways & Limitations

    The complete bilinear ceiling is not directly usable because it has O(C^2d_ϕ) degrees of freedom per edge type.

Abstract

from arXiv · show

Protein structure modeling rests on a single computational primitive: the interaction between what a residue is (sequence content) and where it sits (three-dimensional geometry). What is the expressive limit of this layer class? We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling, while the additive message passing of mainstream geometric GNNs is provably blind to content-geometry binding. We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost: each radius neighborhood is organized into a sequence hyperedge and a contact hyperedge, modulated by an edge-conditioned matrix-valued operator factorized into K learned basis operators with geometry-generated coefficients. Across enzyme function prediction, fold classification, and ligand binding site detection, Hyper-Fold and its hierarchical variant Hyper-Fold-Deep achieve the best results among protein-specific structure encoders; Hyper-Fold-Pocket, an anchored set-prediction head, surpasses UniSite-3D on UniSite-DS and two zero-shot benchmarks with no sequence language model features, 68x fewer parameters, and 4.8x lower latency--suggesting that a sufficiently expressive 3D backbone recovers information that fusion architectures previously borrowed from evolution-scale pretraining.

1 INTRODUCTION

The paper characterizes the expressive limit of sequence–geometry interaction and introduces Hyper-Fold to approach it efficiently. Across function, fold, and pocket tasks, the resulting models achieve strong structure-encoder performance without sequence language-model features.

  • The complete bilinear operator over content–geometry outer products is the sufficient statistic and expressive ceiling for second-order sequence–geometry interaction.This yields an expressivity ladder for comparing interaction layers.
  • Additive message passing is provably blind to content–geometry binding because its content and geometry terms factorize.Binding swaps preserve the additive output even though residue–position pairings differ physically.
  • Hyper-Fold uses rank-K separable matrix gating, organizing each radius neighborhood into sequence and contact hyperedges.The matrix-valued operator is factorized into learned basis operators with geometry-generated coefficients.
  • K=8 captures 90–98% of learned full-operator energy, with accuracy saturating beyond that rank at message-passing cost.This supports an efficient approximation to the expressive ceiling.
  • 0.864 vs. 0.810 Fmax at the 95% sequence-identity cutoff, while Hyper-Fold also improves fold classification and pocket detection.Hyper-Fold-Pocket reaches 0.617 AP@0.3 on UniSite-DS versus 0.560 for UniSite-3D and exceeds it on six zero-shot metrics without sequence language-model features.
  • Hyper-Fold-Pocket adopts an anchored set-prediction head for pocket detection, extending structure-only modeling beyond geometry-tool approaches.The paper positions this formulation within the evolution of ligand-binding-site detection methods.

2 THE EXPRESSIVE LADDER OF SEQUENCE–GEOMETRY LAYERS

This section orders sequence–geometry layers by how richly they bind content to geometry, culminating in the complete bilinear operator. It then argues that low-rank matrix gating can approximate this ceiling while controlling its otherwise large parameter cost.

  • Protein residues carry content features and SE(3)-invariant geometry descriptors, and structure encoders aggregate updated node states over residue neighborhoods.The geometry descriptor includes distance, direction, and orientation terms.
  • The ladder progresses from additive, scalar-gated, and channel-gated interactions to matrix-gated operators and the complete bilinear class.Additive, scalar-gated, and channel-gated layers are degenerate special cases of matrix-gated layers.
  • Additive message passing produces identical outputs for binding-swapped configurations, whereas suitable matrix-gated operators can distinguish them.This establishes the specific expressive gap between factorized additive messages and matrix-valued interactions.
  • Any second-order interaction factors through x_j ⊗ϕ(δ_ij), and the complete bilinear class realizes all such functions.A universal geometry feature map can represent every continuous kernel.
  • The complete bilinear ceiling has O(C^2d_ϕ) degrees of freedom per edge type, making direct use impractical.The paper therefore studies structure in the learned operator rather than deploying the unrestricted ceiling directly.
  • 89.9–98.2% of operator energy lies in the top eight eigendirections, and increasing K from 8 to 16 leaves pocket AP essentially unchanged.These empirical observations support rank-K matrix gating as a small-cost approximation.

3 HYPER-FOLD: MATRIX-GATED CONVOLUTION OVER SEQUENCE AND CONTACT HYPEREDGES

Hyper-Fold organizes each residue’s neighborhood into sequence and contact hyperedges, then uses geometry-conditioned rank-K matrix gating to modulate neighbor content. Its residue-level and hierarchical variants support both local and protein-level representations, while Hyper-Fold-Pocket adds anchored set prediction for pockets.

  • 3.1 SEQUENCE AND CONTACT HYPEREDGES: Each residue neighborhood is split into sequence-adjacency and spatial-contact hyperedges, processed by the same matrix-gated convolution.The hyperedge view organizes neighbors rather than introducing separate hypergraph propagation rules.
  • 3.2 THE FOLD-CONV LAYER: The backbone combines amino-acid embeddings, Cα coordinates, local frames, and jointly conditioned sequence–contact geometry features.Edges carry SE(3)-invariant geometry and clamped sequence displacement.
  • 3.2 THE FOLD-CONV LAYER: GK-Net generates rank-K coefficients from edge geometry, and Fold-Conv applies them through vertex-to-edge and edge-to-vertex outer-product message passing.The rank-K factorization recovers a matrix-gated operator using learned basis operators and geometry-generated coefficients.
  • 3.3 BACKBONE VARIANTS: Hyper-Fold preserves sequence resolution for residue-level features, whereas Hyper-Fold-Deep interleaves pooling with widening channels for protein-level features.Hyper-Fold-Pocket uses the residue-level backbone because every residue must be scored.
  • 3.4 HYPER-FOLD-POCKET: Hyper-Fold-Pocket projects two backbone levels into memory and selects structure-anchored residue queries for transformer refinement and pocket classification and masking.The head outputs a confidence score and per-residue mask for each predicted pocket.

4 EXPERIMENTS

Across enzyme prediction, fold classification, and pocket detection, Hyper-Fold variants outperform protein-specific structure encoders and UniSite-3D on in-distribution and zero-shot evaluations. Ablations and efficiency measurements connect the gains to the aggregation operator while retaining practical cost.

  • 4.2 MAIN RESULTS: Hyper-Fold-Deep achieves the best protein-specific accuracy on the hardest Fold split and also leads on Superfamily, while Hyper-Fold leads EC results.The variants trade residue-level detail for hierarchical multi-scale global representation.
  • 4.3 THE EXPRESSIVITY LADDER IS CAUSAL: OPERATOR-ONLY SWAPS: The operator ablation shows that per-channel modulation closes much of the gap, while full matrix gating leads every metric and gains 3.5 AP@0.5 points over channel gating.The stricter IoU threshold produces the larger margin.
  • 4.4 ANALYSIS: Hyper-Fold-Pocket is 4.8× faster and 68× smaller than UniSite-3D at the best accuracy, using 10.07M parameters and 23.9 ms latency.The comparison uses 683.9M parameters and 114.3 ms for UniSite-3D.
  • 4.4 ANALYSIS: Qualitative analyses associate Hyper-Fold-Pocket’s detection advantage with tighter ground-truth alignment, localized query pocketness, and predominantly true-positive top-ranked queries.The pattern is shown through error maps, pocketness distributions, precision–recall curves, and top-5 query scores.

5 CONCLUSION

The paper identifies the complete bilinear content–geometry operator as the expressive ceiling and presents Hyper-Fold as an efficient approximation. Its reported benefits span protein function, fold classification, and zero-shot pocket detection without sequence language-model features.

  • 5 CONCLUSION: The complete bilinear operator over content–geometry outer products is the expressive ceiling for sequence–geometry interaction layers.Additive message passing is provably blind to content–geometry binding.
  • 5 CONCLUSION: Hyper-Fold approaches this ceiling at message-passing cost and improves results across function annotation, fold classification, and zero-shot pocket detection.The conclusion reports these gains without sequence language-model features.
  • 5 CONCLUSION: The experiments use public protein benchmarks and involve no human subjects, sensitive data, or foreseeable harmful applications.The authors state adherence to the ICLR Code of Ethics.
  • 5 CONCLUSION: Anonymized source code, configurations, and trained weights are included, and the pocket model runs on a single PDB file out of the box.Architecture details, training recipes, and proofs are provided in the appendices.
  • 5 CONCLUSION: The work positions its layer-level expressivity analysis as distinct from broader graph-network and hypergraph expressivity characterizations.Related work describes the analysis as finer-grained within geometry-conditioned protein layers.

B ARCHITECTURE DETAILS

The architecture uses sparse Fold-Conv blocks with rank-K geometry-conditioned operators, multi-scale backbone readouts, and an anchored transformer pocket head. Detailed configurations preserve linear-length scaling while specifying proposal selection, positional encoding, matching, and denoising procedures.

  • B ARCHITECTURE DETAILS: Fold-Conv stacks residual blocks whose geometry builder computes edge features, GK-Net generates rank-K kernels, and outer-product messages aggregate over two hyperedge types.The implementation uses a bottleneck pre-activation residual unit.
  • B ARCHITECTURE DETAILS: GK-Net uses sequence-indexed buckets and a smooth gate that downweights edges far in both space and sequence.The shallow and deep stages use different bucket counts and radii.
  • B ARCHITECTURE DETAILS: Rank-K kernel evaluation costs O(|E| K C C′), matching additive message passing’s O(|E| C2) order up to a small constant factor.Sparse neighborhoods make the per-layer cost grow linearly with sequence length, unlike all-pairs transformer attention.
  • B ARCHITECTURE DETAILS: The pocket head projects two backbone levels into query, key, and memory features, selects top-Nq residue proposals, decodes them, and predicts scores and masks.The proposals are structure-anchored through residue features and Cα coordinates.
  • B ARCHITECTURE DETAILS: Anchors and memory receive translation-invariant 3D Fourier positional encodings, while invalid and unmapped positions are excluded by a padding mask.The backbone features are scattered onto the UniProt residue axis with a learned null vector for uncovered positions.
  • B ARCHITECTURE DETAILS: A 4-layer decoder cross-attends from anchored queries to bi-level memory, and each query independently produces a pocket probability and residue mask.Mask binarization uses 0.5, and the pocket score is calibrated by mean mask probability.
  • B ARCHITECTURE DETAILS: Hungarian matching combines classification, binary cross-entropy, and Dice terms, while contrastive denoising adds clean, lightly noised, and heavily noised queries.The proposal count is Nq=50 and denoising queries are capped at 32.

C EXPERIMENTAL SETUP

The experiments use unified training protocols across EC prediction, fold classification, pocket detection, and latency benchmarking, with task-specific schedules and hardware controls.

  • Baselines and provenance: Classification baselines include GCN, GAT, Point Transformer, SchNet, GVP, GearNet variants, ProNet variants, CDConv, and SCHull.Most baselines follow a unified recipe, with documented deviations for Point Transformer and reference-build measurements for GCN.
  • Classification training: EC and fold-classification models use AdamW with a 10^-4 learning rate, warmup, cosine decay, batch size 16, and 300 total epochs.EC additionally uses EMA, mixed precision, and gradient clipping; fold experiments also include an SGD recipe and structure-forcing augmentation for Hyper-Fold variants.
  • Pocket training: Pocket models use AdamW at 2×10^-4 for 40 epochs, with warmup, cosine decay, EMA, gradient clipping, and Gaussian coordinate noise.Weight decay is applied only to parameters with at least two dimensions, and the coordinate-noise standard deviation is 0.2 Å.
  • Latency measurement: Latency is measured on a single A100-80GB at batch size 1 using a synthetic length-300 protein and timed fp32 forward passes.The protocol uses 10 warmup and 50 timed passes; graph construction is included or excluded according to each model’s native pipeline.

D.1 FULL EC RESULTS

Hyper-Fold is the strongest EC method across all five sequence-identity cutoffs, with its advantage increasing on more remote-homology cases; Hyper-Fold-Deep remains ahead of every baseline.

  • EC benchmark: +2.1 Fmax points at the 30% cutoff gives Hyper-Fold 0.726 versus 0.705 for the strongest baseline.The advantage is +1.1 Fmax points at the 95% cutoff, where Hyper-Fold scores 0.864 versus 0.853.
  • EC benchmark: Hyper-Fold is the best method at every sequence-identity cutoff, spanning 30%, 40%, 50%, 70%, and 95%.Table 4 also reports micro-AUPR at the 95% cutoff.
  • EC benchmark: Hyper-Fold-Deep trails Hyper-Fold slightly but stays ahead of every baseline at all five cutoffs.This preserves the performance ordering across the complete EC cutoff sweep.

D.2 ARCHITECTURE AND GEOMETRY ABLATIONS

Ablations identify shallow geometry schedules and channel width as important design factors, while the rank-K kernel provides the preferred route to expressivity under the shared EC setup.

  • Architecture and geometry ablations: 8 Å shallow-layer radii match the default closely at Fmax@50, while shrinking them to 6 Å is harmful.Uniform Radii versus default scores 78.55 versus 78.87 Fmax@50; the comparison is made under the same EC benchmark.
  • Architecture and geometry ablations: Only the quarter-width variant departs markedly from the default in the EC architecture and geometry heatmap.Figure 5 compares each variant’s test Fmax against the default at the same sequence-identity cutoff.
  • Architecture and geometry ablations: 4.6 Fmax@50 points separate the quarter-width model from the default, with scores of 74.25 versus 78.87.The bottleneck C/2 design matches full width, indicating that the quarter-width configuration is the marked width failure.
  • Architecture and geometry ablations: The default schedule is strongest at the strict 30% and 40% cutoffs, whereas contracted shallow radii with a longer deep-layer window trade strict-cutoff accuracy for lenient-cutoff gains.The ablation varies per-layer radius schedules and sequence windows while sharing the six-layer Hyper-Fold recipe.

E.1 POCKET ABLATIONS

Pocket ablations show that rank, multi-scale readout, contrastive denoising, and content–geometry binding chiefly affect strict-IoU performance, with the selected design balancing accuracy and cost.

  • Rank K: +4.4 AP@0.3 and +9.4 AP@0.5 accrue from increasing rank K from 1 to 8, after which accuracy saturates.K=16 adds nothing on AP@0.3 and 0.4 points on AP@0.5 despite 31% more parameters and 25% higher latency.
  • Readout: The two-scale {3, 6} readout is retained because multi-scale variants stay within noise on AP@0.3 while the default has the best AP@0.5 at lowest cost.The {3}-only readout collapses, while {6}-only matches AP@0.3 but trails on AP@0.5.
  • Contrastive denoising: Removing CDN costs 0.6 AP@0.3 and 1.5 AP@0.5 at identical latency.The larger strict-IoU loss is attributed to contrastive negatives teaching the decoder to reject near-miss decoys.
  • Cross-ablation pattern: Across all three ablation groups, design choices move AP@0.5 about twice as much as AP@0.3.The strict-IoU metric is therefore the more sensitive axis for these pocket-head design choices.

E.2 QUALITATIVE ANALYSIS PROTOCOLS (FIGURE 4)

The qualitative analysis compares Hyper-Fold-Pocket and UniSite-3D using standardized residue-mask errors, query visualizations, and pooled precision–recall evaluation. The accompanying expressivity proofs establish additive message passing as a restricted matrix-gated case and complete bilinear interaction as the second-order ceiling.

  • Error maps: Error maps compare ground truth, UniSite-3D, and Hyper-Fold-Pocket masks using TP, FP, FN, and residue-set IoU.Hyper-Fold-Pocket uses the top-five query with highest IoU against the ground-truth mask union; UniSite-3D uses its highest-scoring prediction.
  • Query constellation: Query constellations place each nonempty predicted mask at its Cα centroid, with sphere size encoding query score and color encoding pocketness.The visualization shows all 50 decoder queries from one P00359 forward pass.
  • PR curves: Pooled precision–recall curves concatenate and globally sort query score, TP, and FP records across all test proteins to compute AP.The curves use all 50 queries per protein at IoU thresholds 0.3 and 0.5.
  • Score histogram: The score histogram instead summarizes the top-five query scores per protein, separating true and false positives at IoU 0.5.This operating point differs from the all-query PR-curve evaluation.
  • Expressivity analysis: Additive message passing is a matrix-gated layer with a geometry-affine kernel, whereas complete bilinear interaction represents all second-order content–geometry messages.The proof hierarchy places additive MP inside matrix-gated layers and matrix-gated layers inside complete bilinear interaction.

F.5 A SPECTRAL TAIL BOUND FOR RANK-K GATING

The spectral analysis bounds rank-K operator approximation error through the eigenvalue tail of the learned operator family. Measurements show that rank 8 captures most operator energy, while increasing to rank 16 provides negligible pocket-detection gains.

  • Operator spectrum: The learned full operator family is analyzed through the second-moment matrix of vectorized edge-conditioned operators and its eigenvalue spectrum.Each eigenvector is reshaped into a basis operator for the rank approximation.
  • Spectral bound: The optimal rank-K approximation achieves the spectral tail error, with equality for eigenbasis operators and orthogonal-projection coefficients.Ky Fan’s maximum principle selects the top-K eigenvectors as the optimal basis.
  • Deployed rank: 89.9–98.2% top-8 energy per block implies at most 10.1% relative operator error in every block and at most 8.2% in blocks 1–3.These figures are measured for the deployed rank-8 factorization.
  • Realizability gap: The spectral tail is a lower-bound floor rather than achieved error because the bucketed weight network restricts the coefficient functions.The remaining gap can shrink with network capacity and is therefore not a structural limit of the factorization.
  • Measurement protocol: Operator spectra are collected from a deliberately over-parameterized K=16 model across edges of 64 validation structures and evaluated by top-8 eigendirection energy.This measurement estimates how many gating directions the learned model uses relative to deployed rank 8.
  • Accuracy saturation: 31% higher parameter cost for increasing K from 8 to 16 yields no AP@0.3 gain and only +0.4 AP@0.5 points on pocket detection.The K-sweep indicates accuracy saturates near the rank selected for deployment.
Loading 2608.29207v2…