Source-linked AI summary
Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks
Osvaldo M Velarde, Lucas C Parra, Alireza Hashemi, Hernan A Makse
TL;DR
The paper asks how to understand neural networks whose internal structure remains opaque despite empirical scaling. It analyzes learning through graph symmetries, proves that covers are stable attractors of stochastic gradient descent, and uses symmetry for compression and continual learning. The work also identifies open boundaries involving compression pipelines, dynamic compression, and formal links to data hierarchy.
Problem
Neural networks remain poorly understood internally, while scaling produces costly, data-inefficient, and sometimes less plastic systems.
Method
The paper develops a mathematical framework for fibrations, opfibrations, and coverings, detects them with balanced coloring, and manipulates them through compression and symmetry breaking.
Results
Covering symmetries are stable attractors of stochastic gradient descent, and the framework supports model compression and continual-learning performance improvements.
Takeaways & Limitations
Emergent symmetries provide an interpretable colored-graph view of learned regularities and a theory-driven route toward more efficient inference and lifelong learning.
Takeaways & Limitations
The approach currently relies on static post-training compression, and preserving both forward and backward computations requires all covers to have the same cardinality.
Abstract
from arXiv · showhide
Artificial neural networks are often regarded as powerful yet opaque black boxes. Here, we demonstrate that learning in deep neural networks generates local symmetries known in graph theory as fibrations and coverings. We prove that covering symmetries are stable attractors of stochastic gradient descent. Consistent with this theory, we report the emergence of covering symmetries across major network architectures, including multilayer, convolutional, recurrent, and transformer networks. Exploiting these symmetries enables drastic model compression - reducing networks to 17% of their original size without sacrificing performance. Furthermore, controlled breaking of covering symmetry overcomes the loss of plasticity, achieving state-of-the-art performance in continual learning. The theoretical results provide a new foundation for AI systems based on symmetries that convert black boxes into interpretable colored graphs and enable more efficient inference and lifelong learning.
1 Main
The paper introduces graph-based local symmetries to explain neural-network structure beyond opaque scaling-driven models. It develops fibrations, opfibrations, and coverings as a hierarchy that supports synchronization, compression, and symmetry breaking.
- Scaling-driven development has increased model size while leaving neural-network internal structure poorly understood.Large sequentially trained networks also face high cost, data demands, and loss of plasticity.
- The framework generalizes rigid global symmetries to local graph symmetries called fibrations, opfibrations, and coverings.These structures arise in the input and output trees of computational graphs and form a hierarchy of increasing strictness.
- Symmetry breaking for continual learning first compresses the network to a covering base, then randomizes or zeros redundant weights to create new degrees of freedom.The figure presents this as a two-step procedure for preserving the learned task while enabling continued learning.
- Fibration fibers group nodes with isomorphic input trees, identified by balanced coloring that assigns the same color when nodes receive the same set of input colors.The coloring process provides an operational way to detect local symmetry classes in network graphs.
- Fibration compression merges same-colored nodes into a smaller base graph while conserving input trees and preserving the original forward computation.This yields a network with fewer effective degrees of freedom and can reduce model size without losing performance.
- The hierarchy culminates in automorphisms, but the paper reports observing less restrictive local symmetries rather than automorphisms in trained networks.Automorphisms preserve the entire graph connectivity, whereas local symmetries impose less stringent conditions and are more common.
2 Results
The paper establishes that deep learning forms hierarchical local symmetries, with synchronized activities, errors, and weight updates, and shows that these structures support compression and continual learning across architectures.
- Theoretical results: Deep learning organizes networks into hierarchical local symmetries—fibrations, opfibrations, and coverings—that emerge across MLPs, CNNs, RNNs, and Transformers.These structures generalize beyond ordinary parameter tuning to weighted computational graphs and operations including convolutions, gating, residual connections, and attention.
- Theoretical results: Nodes in the same fiber synchronize activity, while nodes in the same flavored opfiber synchronize backpropagated error signals.For quasi-fibers, synchronization accuracy is bounded by εfib; quasi-opfibers use an analogous threshold εop.
- Theoretical results: Covering partitions are preserved under gradient descent, so covers can merge but cannot split and therefore form stable attractors under stochastic gradient descent.The mechanism follows from synchronized learning, which makes corresponding weight updates identical.
- Compression: Covering compression reduces a three-layer MNIST MLP to 17% of its original size with no change in loss or accuracy, while generally outperforming fibration and conventional pruning.The compression merges nodes with identical functional roles, whereas pruning can remove unique contributions and immediately affect loss.
- Continual learning: Fibration Symmetry Breaking restores degrees of freedom after cover compression and reaches 95% accuracy by task 5,000, reducing the remaining deficit versus continual backpropagation by half.The method continues to outperform continual backpropagation beyond task 1,500 in the sequential ImageNet task.
3 Discussion
The framework interprets learning as structure formation through emergent local symmetries, supporting compression and a controlled stability–plasticity trade-off in continual learning.
- Discussion: Optimization stabilizes covering symmetries and aligns network topology with learned data structure, producing interpretable colored graphs.The authors connect synchronized activity and redundant nodes to emergent fibration symmetry in deeper over-parameterized layers.
- Discussion: Weight-tying and related inductive biases can be understood as graph symmetries that may emerge through learning rather than being hardcoded.
- Discussion: 50% reduction in the performance gap on sequential ImageNet benchmarks is achieved by Fibration Symmetry Breaking compared with continuous backpropagation.FSB compresses emergent covers to preserve prior knowledge and injects randomized nodes to restore degrees of freedom.
- Discussion: Dynamic compression during training, optimal sequencing with low-rank and quantization methods, and the link between symmetry quotients and data hierarchies remain open.
- Discussion: The framework presents learning as structure formation that exposes learned regularities while offering a path beyond brute-force scaling.
4 Methods
The methods distinguish global automorphisms from local fibrations, opfibrations, and coverings, and represent these symmetries through hierarchical balanced colorings and graph partitions.
- Definitions: A covering symmetry is a surjective homomorphism preserving both input and output trees, whereas an automorphism is a bijective global permutation preserving adjacency.
- Partitions: Fibration, opfibration, covering, and automorphism symmetries induce fiber, opfiber, cover, and orbit partitions ordered by coarsening.
- Balanced colorings: Fibers and opfibers are in- and out-balanced colorings, coverings are both in- and out-balanced, and orbits additionally require an automorphism permutation.
- Global and local symmetries: Fibrations, opfibrations, and coverings relax automorphisms’ global constraints by preserving input, output, or both local trees.These local symmetries form categories rather than groups, while color-preserving tree isomorphisms form groupoids.
- Example: Nodes 9 and 10 form a cover because their input and output trees are isomorphic, although they are not in the same orbit because other graph adjacencies are not preserved globally.
- Conventions: The formulation assigns distinct balanced colors to input-layer nodes and distinct opfiber colors to output-layer nodes, differing from the original graph-fibration definition.
- Flavored graphs: For flavored connections, tree isomorphisms must preserve both node and connection mappings together with connection flavors.
4.3 Fibration lifting operation for weighted network
Fibration compression merges nodes with matching weighted input structure into smaller bases, while lifting expands a base non-uniquely and preserves the network’s forward computation.
- Lifting: Compression is non-injective, so lifting a weighted base back to a full network is not unique.Different full-network configurations can share the same base through intermediate lifting dimensions.
- Functional equivalence: All four example networks and their bases have identical forward outputs and loss values for every dataset input when they share the same minimal base.
- Weighted fibers: Two weighted nodes share a fiber exactly when their total input from every previous-layer color is equal, with input-layer nodes assigned distinct colors.
- Synchronization: Fibration symmetry induces synchronized activity during forward inference, linking weighted graph structure to network function.
- Symmetry hierarchy: Fibration and opfibration symmetries are weaker than automorphisms because they impose less restrictive weight constraints.
- Compression: Fibration compression merges nodes sharing a fiber while aggregating incoming weights to conserve their summed effect.
- Opfibers: Opfiber construction aggregates outgoing weights over existing opfibers and preserves connection flavors through an equal-sum criterion.
4.6 Proof of synchronization in Eqs. (15) and (17)
The proof establishes forward activity synchronization by induction across layers and shows that, for linear activations, synchronization across all inputs identifies fiber membership.
- Forward synchronization: The synchronization proof assumes activity synchronization at layer ℓ−1 and extends it to layer ℓ by induction.
- Forward synchronization: The base case at layer ℓ=1 completes the induction proving Eq. (15).
- Inverse relation: For linear activation functions, synchronized activity across all inputs implies that two nodes belong to the same fiber.This makes the relationship between fiber structure and activity synchronization invertible in the linear case.
- Backward synchronization: Error synchronization is proved by applying the backpropagation rule inductively from layer ℓ+1 toward layer ℓ.
4.7 Quasi-synchronization of activity and error in quasi-symmetries
The paper characterizes how activity and error synchronization relate to fibration and opfibration symmetries, with exact equivalence in linear networks and more constrained behavior under nonlinear activations.
- Quasi-synchronization: Quasi-fibers and quasi-opfibers admit upper bounds on activity and error variation when the activation is Lipschitz.The bounds use the activation’s Lipschitz coefficient and thresholds εfib and εop.
- Linear equivalence: In linear networks, synchronized activity implies fibration symmetry, while synchronized errors imply opfibration symmetry.The forward and backward arguments are analogous when activations are linear.
- Nonlinear coupling: Nonlinear backpropagation couples error gradients to forward activations through σ′, requiring flavored opfibrations to represent error-flow dependencies.Connection flavors inherit forward color assignments and preserve them in the output-tree structure.
- Linear equivalence: For linear activations, synchronization and (op)fibration are bidirectionally equivalent, yielding a strict structure-function correspondence.This equivalence relies on σ′ = 1 and the decoupling of forward and backward passes.
- Alternative definitions: Alternative cover definitions replace weight-based flavor conditions with synchronization requirements or recursive structural constraints, but activity-dependent variants lack an obvious weights-only coloring algorithm.The recursive definition couples opfiber and cover structure because fibers ensure activity and slope synchronization.
4.10 Covering symmetries emerge during stochastic gradient descent
The paper proves that covering relationships are preserved by gradient descent: synchronized updates keep nodes within covers, while distinct covers may merge. Under suitable conditions, these invariant sets become stable attractors of stochastic gradient descent.
- Cover definition: Covering symmetry combines membership in the same fiber with membership in the same flavored opfiber.This defines the covering partition used in the stability analysis.
- Cover stability: Nodes in the same cover remain in that cover after each gradient-descent update, although distinct covers can merge into coarser covers.The result establishes cover invariance rather than permanent separation between different covers.
- Stability rules: Fiber formation propagates forward from the input, whereas opfiber formation propagates backward from the output during learning.The stability rules connect current covers to fiber and opfiber structure in adjacent layers.
- Cover stability: The cover-preserving parameter sets are invariant under gradient descent updates.This invariance is the bridge from the coarse-graining theorem to stochastic-gradient dynamics.
- SGD consequence: Covering symmetries are stable attractors of stochastic gradient descent when the learning constant is sufficiently large.The cited result attributes attraction to damping of stochastic fluctuations transverse to the symmetric set.
4.12 Fibers coloring for sums, convolutions, gates and concatenations
The paper extends fibration coloring and compression beyond feedforward layers to sums, convolutional features, recurrent gates, concatenations, and transformer attention, while leaving time, pixel, and token symmetries for future work.
- Sums and concatenations: Sums and concatenations can be handled by combining their inputs and preserving the colors contributed by each input source.Residual connections are treated as sums, while concatenated vectors contribute their own node colors.
- Gates: In LSTM gates, fibration compression applies to multiplicative interactions, with corresponding gate structures sharing fiber and opfiber roles.The figure’s base uses 2 nodes per gate as an example of aligned compressed gate structure.
- Convolutions: Convolutional fibration compression reduces feature-channel counts, exemplified by changes from 3 to 2 and from 5 to 3.The figure treats features as nodes whose channels can be merged according to fibration structure.
- Attention: In attention modules, fibration compression preserves the number of interactions even when the query and key have different numbers of fibers.The illustrated attention example retains 3 multiplications.
- Scope: Symmetries in time, pixel, and token dimensions are outside the paper’s current treatment and are left for future work.The coloring discussion instead focuses on feature channels and related node dimensions.
4.13 Fibration compression rules
Fibration compression merges nodes with matching input structure while preserving forward computation, and extends to opfibrations, CNN channels, recurrent gating, and attention interactions. Approximate symmetries are detected through tolerance-based colorings and can be computed efficiently across layers.
- Parameter rules: Base-network parameters add weights sharing input colors and average weights sharing output colors; opfibration compression preserves backward computation.The feedforward and backward colorings are generated in opposite directions.
- Other architectures: Fibration definitions generalize to CNN channels and computational hypergraphs with multiplicative gating, including LSTM gate products and transformer attention interactions.For gated interactions, matching fiber classes preserve the corresponding product activity.
- Other architectures: LSTM fibration compression reduces gate-layer dimensionality consistently but preserves the number of interactions in the product layer.The forget, input, and product layers compress to a common reduced dimension, while q and k reductions do not reduce z interactions.
- Approximate colorings: Approximate continuous-weight symmetries are found by replacing exact equalities with tolerance-based inequalities and clustering nodes using a distance threshold ε.The resulting quasi-fibers and quasi-opfibers may use shared or layer-specific thresholds.
- Algorithmic cost: The coloring algorithm’s operation count scales linearly with parameter size within each layer and linearly with the number of layers.This supports efficient analysis of layered feedforward networks.
4.16 Optimal quasi-cover thresholds for a given loss threshold
The paper chooses layerwise quasi-fiber and quasi-opfiber thresholds by maximizing compression subject to a preset allowable loss increase. Gradient-based estimates avoid repeatedly evaluating loss on large datasets, while the accuracy–loss relationship remains task- and architecture-dependent.
- Loss estimation: Gradient-based estimates approximate compression-induced loss change using training gradients that are independent of the threshold vector.Only the compression-induced weight change varies with the selected tolerances.
- Constrained optimization: The optimization maximizes compression by minimizing the collapsed network’s parameter count subject to a loss-variation threshold εLoss.The threshold vector contains fiber and opfiber tolerances for all layers.
- Threshold selection: The acceptable εLoss is defined where further loss reduction no longer improves accuracy, and its value depends strongly on architecture and task.Accuracy may stabilize while loss continues decreasing during training.
4.17 Redundant feature pruning is a special case of fibration compression
Duplicate-feature pruning is a restricted covering symmetry, while the paper links symmetry formation and symmetry breaking to continual-learning behavior. Orthogonality prevents fibers, whereas breaking covers broadly targets the structures associated with reduced plasticity.
- Redundant feature pruning: Removing nodes with identical input and output weights is a special case of covering symmetry corresponding to a trivial single-layer node permutation.The broader fibration framework allows approximate or summed weight relationships beyond exact duplicates.
- Symmetry prevention: Orthogonal regularization prevents output-layer fibers because normalized orthogonal weights make the clustering distances separate all dimensions.Under this condition, no output fibers exist.
- Continual learning: 88% early-task accuracy can decline into loss of plasticity by task 2,000 under unmodified backpropagation, a pattern reported across architectures, parameters, and optimizers.The experiment used sequential binary ImageNet classification tasks.
- Continual learning: Large covers emerge during sequential ImageNet training, while continual backpropagation and symmetry breaking keep cover sizes smaller and their numbers larger.This pattern appears across the first task’s 250 epochs and more clearly after 900 tasks.
- Symmetry breaking: The proposed symmetry-breaking protocol targets all covers, unlike deep continual learning, which resets only inactive covers with zero utility.Breaking both input and output structure changes the nodes’ trees and breaks their associated covers.
4.21 Fibers are a refinement of clustering partitions
Fibers refine synchronization-based clustering partitions and are linked to hierarchical cover formation under SGD. These structures also motivate compression and exhibit an invariant-subspace mechanism in learning.
- Class-conditional synchronization clusters group nodes whose activity is synchronized for every input in a given class.
- Fibers refine intersections of class-conditional clusters because nodes in an intersection synchronize across the dataset, though not necessarily on unseen inputs.
- For small thresholds, fiber partitions and synchronization clusters perfectly match under random inputs, with matching score=1.
- Covers refine fibers and merge hierarchically during learning; once nodes join a cover, the transition is irreversible.
- All network configurations sharing the same minimal base have the same loss value, producing a hierarchical loss-landscape organization across lifted networks.
- If a layer's input lies in a linear subspace, SGD keeps the input weight matrix in that subspace, making it an invariant set and stable attractor.
Declarations
The declarations report funding, no competing interests, code availability, and author contributions.
- Partial support came from the National Institutes of Health and Army Research Office.
- The authors declare no competing interests.
- The code is available at Github/MakseLab/fibrations in dnns.
- Authors contributed to conception, design, proofs, writing, data analysis, and analysis-code development.